navidrome: a music server on its own box, with beets in front of it

Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.

── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.

Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.

The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.

A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.

── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.

── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.

/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.

── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).

/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.

── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".

inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.

50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.

No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/group_vars/all/main.yml

@@ -57,6 +57,11 @@ subdomains:
   # unauthenticated, like any block explorer - the data is the public chain.
   monero_explorer: exmrplore
 
+  # Navidrome music server (music-box, proxied via vipy). No basic auth:
+  # Navidrome has its own login, and the Subsonic API the mobile clients speak
+  # has nowhere to carry a second credential. Same call made for monero above.
+  music: music
+
 # Read by plays across several groups, so global rather than group_vars/<group>.
 headscale_namespace: counter-net
 

ansible/group_vars/all/vault.yml

@@ -1,191 +1,214 @@
 $ANSIBLE_VAULT;1.1;AES256
-61616634393062353739616362333531616534666431633234333539376337313235366337346364
-3135643764366262633636353735613233343361363165390a303638313465643332363839326530
-66623032363563306135643838626435646639306438393638353133393731386139336661383461
-3965353830373536360a653364663734333762616630313563656335333933333566333334393837
-64636438386264316333633839623736333932356365643639376262353564363639303438373534
-37346139393038643332626337646430376165316133343362636235623838343637343062623431
-35333530376430643562656233356165313065633865343439363238663966366332346536363430
-37313261363434376438306333373161303465653132353864366562363235376630373262356633
-64393830643037346565303435633966303234666638653833353138346464383462383136313433
-38356663303231313134623862643762633762303364616433623963326463326232333662643839
-35346435643234663838643934376461393631666363333435623965376239353137613533383033
-34663662666362613661613833383166343965333939336631363633343332653064653761353063
-31356263396434386433393332353532373663396431313539343138623564393434373135616239
-39353163363063323064633935346134663165626265633136633833643538616662333663366664
-61323866313266626637356532336166643261343662636464346566613531626532663961613238
-65353434653466316438343631376332316566376438623835393438313239326339323564623564
-37313938396562623865646338336634333637666538656139646539376531656464653136663462
-30343537303230306331386139376261396162326437623265373336356163636562623932653562
-64353763646636363839613365386233316262356339653332323565383033613561356236666336
-39616636353530636434366430643239383164376139343064316430396431363363383166346130
-61373735313633313639323933646632393231633935326439316466623438393233623139626139
-38353333376164613765656164363436613831663333656239346661623031393238353765356233
-65336632633361356432326339613766633938336537306261633236333835646335356365393330
-38656236363864613862643365383265633134643338383033643434376530303462336564383066
-66663061646432333034316431353236323938623639303263393537313939373162633838343062
-32316230613033363931343162386538376435386139396634333061313930373335306663393938
-30643466303338316663613563636239343432363964653438666662356437623962653266346434
-64616136643835396430306633313736623736316634313466366538663437633437613963386231
-39666565366464326132643836366431653535343162336563356664346166313532393664353139
-39383037323234323038316632383766383266393663646438653038643566376133636237653733
-38623438343432323664303461373930363437356136333235393262336336633266396531663638
-30393335396537356635343730376532363261353039396639653966633033383037656536646161
-39653531303962656234343634346236373362616634313664613764373439643734306639386461
-38663364613365643030616430356139393362386438613838326236643063633538346262626133
-33666537353862353461393939656231666330633434316261656134363062353835663833356435
-62663037383266353037326431376438313165343231656366306265666464303132613133303662
-39646462333439633065356137633932336162333438356337663637353033326362313036303564
-38643662633531383539393064356535363631336266316334386465313965376665306163393664
-64643434326561613333353134633166613464393537323263656466613239343562653636353764
-36343365656636393031626231343033653764393361376532646139613166306561323364356364
-62623838316430616636323966616634343732353031393665356137393532383762363565663038
-35323930633933626464633966343865346466353465626466663435316137366435373734363661
-33663939366165313939643032396562326565343961376366313730333735336333353164623266
-30306261363230306562343433656133303662306563646166386531386439633534623632373932
-30623361346632346463333661366539613233383337353961383564303438643965636235396163
-63363264383762303631613033663039626539336566653062333632383534306230396139613234
-62633862643535383536353537393431376431303262323332613731636534643033303635613030
-61653234313732643563383639303633343266623761623435373734613235656263353864353434
-63353039386665623164626666626330623730616438373832303337633364633735613364623131
-35613433366361616239363464343937363566383934313863636231373434636331383939623566
-34623933373434346561633331653163323664393130336665393465303162396630376236653262
-39393465346632323435636333613361353539643238313239373231376436313631663065386262
-34663637386632396137353261613666316564333636326231636137613235623765313137306537
-38306161633731343838356635613038616363653833623634633061376364376534323532666337
-65366634623331323562323562613735356235393434323063303766653334363236613036353638
-35353438623930366336383062303462646338633939393639643832623730633562306338313939
-35666334336334653534646662316534376235656364356563316632363130356635333336666130
-34616639393834373362323930613863383833663161316339323465616639633531376232303332
-33333333643934623538633738643631386232643932303138383861313261306261323532393233
-61353431373538313762653839343861313337643633643032333765396466393831393162306436
-32393832363638333832666364646266376634663031333932333735643831623361323564386261
-61323937393762613531646564363963626461343262373839383938663661626562613837323133
-39646632316666396637646664356230356131313062363039656163613238386334636636636636
-38363463633733663132366239343766353038383962353639313562343636623261376566653635
-35326338663935386132333035373362326235643864336162303063623539656439306466663834
-38333362353630363835613464303131306333313865623236633362623832633062333565343762
-39336161376661646238313839633165666435346433336337633734353166636566326464666330
-37386536613430343139653134343963626466333734356461323631623761663639633532376133
-61363632303566336161653430613735316562656334323265303163383939323564356465636530
-61626262653461386332646566306666323434613162613033373730303265306231316630346239
-39343735363632623561373838316334393166303365363766663965306161643365656132653531
-32613662663362653366633237386362623962323230303538633732663061376334393666326535
-35653038306233626133363732393964623630383637363937363335663836346564636639383266
-32326433316236376363333163346234303938626635363464636563653032656361323138326530
-36636163356632613738626136376563656233313034666161323362313533366461326132316162
-61646335383662313234666332623932396632333435316466346538313630633038663231373764
-39343933613361336439636536356136383535336262613538323837633563633038306231663632
-33396366343466383538306638656234383930306539633865373031343030333765663735386438
-35363232616533656632396661386566643738353862646535323533323237636137646239353664
-32353261366334353933376436336533306531356462343535346664656435626466356662613032
-32656666336235326165653861326431653335653131663034386166373434303936646634366635
-38356536343664363836616235326465616131636132643331373366656164653638613066333737
-38373864333734306234383130376239373238636262376664613865363265363337393031616566
-35336436373535613535653330306164373462313839316636326534366163326538653633376337
-34643039373461393565653838323061656330346632316133373064613661306365336462646237
-62326134623063346431643335313166306461653239303537663132643239386636653363323662
-39393262613766343563636561356362653665313234366532626638346264316363343264313435
-36386531656534666462383337343835313638376438653165663839393433376332623938363738
-35356538383136333534386531376530383936633463363166326133303264373534613066356538
-34373637303333333836386238303061303834383331653735336233666164656530656666356533
-39343030643236613831663465323735343737623362383734326162356333323864333035366433
-63316264613337363136613062663263313731636335383531383933623339666134643539353330
-61333330336132636666383561356565346563393937653539373431653561653664616137623133
-39336333313330336137346238306130653730326130323163306464376434353736623232656134
-33356435663762666266393461363531396334313733333132666363316133346531626564626334
-66356436393062373364316439646235666336333039383137353764303763636461663638333362
-66306430396530636538353965333862323363323832373033353530306331653935396134323634
-61353964303666643163646464613737393863373236613532353761636230336633373461343266
-34383031353734346465613030653639656366313530333265383632343465393566386161373633
-34313563636265336239323362316432383637303134376331633039333537613461353135303934
-66366334663032336330386632656162643933316666643562616564613633303231666536653536
-61376534663632316338653231653561643338356661353232633431623834396635383034366438
-30376161663339633530646539313132383536303931333366316631356135303933316162373730
-38343139653932666335353437613734366530616261343134386261313330303766666136616532
-38663635393236376661623862646265626239656665323332633532616536643237333137343763
-37663432366337633662303931353939346462633837346638353530636132313436626161633865
-30653065666261363238366534656265663039373962333938633335373163383261363434646437
-35356538623765343866653065306164363363616137323065333062386435393539303930363239
-62653638303766343738346437323536306139303261393863373035336662316566646565316662
-30633464663666383135663365636436363132363437633933643464626339643865383066623039
-66356338633939386337353135613066343236356262363164666466333462383364663134616433
-38626332623530633765656464383136383466623561633666366566316461356437626165316166
-65626339383133623439366662373666346539613231306564643731383935633939343865613534
-66363934373433363134323165383830643535613139313139316137643532303562393432383330
-61376363363939633835383161396464613662383961366634626463346430303631653930613430
-66616132616364623261363731343437633032383361356433333531363964633962336338373338
-61353134616332663135663737613230666534656232623933613862326366393538373737393838
-30343037343432306437663035613831316432336431383938363338316437616164376435376261
-66363664333031653962383630656430646238353337616535343535373935663561306536353165
-65343235396537366635633638306135383164353562336331376336323633323338353361353061
-65623431633731303936303732613964346139306336356461626466343132383162666337333664
-38636631636338353339316666643436386537336231353332663636383337343831323939396636
-30663634653733623530343537636137623332666666343433643830333737336639313064386333
-34656362613863323262326130666361373434336234366337663236663739383366373635343862
-62656365366533633263323963346363396237393462653633313133353935343631393032303566
-66326266353264306266613936333734643631376637663137373232653962646534643565326366
-64646634346534613730646430333432373464653365663666346232373332346231306563636136
-38646565366661613161383530326261613537366633323239656461343337363139353164653935
-66356636653232643035363564306438396233376465646462316463333832636361363137363364
-66653164363836333261386437373864666438383338383963616662386631306530326535303666
-37393037313539393535633333663662363164323635333635646638663264383533303032303561
-34613563353237633761363435636138616462656263353966636363326533363235663761643033
-63613934323330363732353465316261653964656430383532623833653663306632633832653763
-32333932663362663664663535363566656665653839386363373066343336353264663136326136
-66383461626137353430316266356366393937613234656261646632366130613739353266613163
-30383731373137653137353663643866343632383937613136666361626435393331313130386230
-33306439633138306331616165386239643765323765366530336337623264343030333234336234
-64333333613838333734346361636162346564656334396139373535333830313037383864623561
-38336131343532663761326635633531323766363963663763653630643232663631613465333132
-35306635636437323835326633616665313133393162623737386538373364343865383634636463
-33626165623962656432396237653665346464663066633633633564313266343662336333656166
-34646636663864663439356364383031313065663433386530663365353937626366663334346434
-35656365316134306161656133653736306537613133386634386638376534376133346237353765
-64363964303535653037323566653838633231363132666237353765643137383030623930393931
-33313935663366313362656137613865386532303737306437353437336331643435323030663666
-66383139313333633538373836396263353337316264656631376236346466353765653830366366
-66386131336431336466636166366139396337386135636262316237306561353563623061313266
-33343563393031303133393163653638643065636636653766613836386438626332373965383864
-36343936613738336132396265663130643739373264356530636363633239346637653136343761
-64646137383235636231313736343162353237636566663830373739636536393864646639343564
-61323039376537646531633030626665323537366233353939653665326666653263623431356166
-34343538613031336561303165323134303862666139633937323331643930383938666138346230
-33633537666431643234336134333333633664333564396139363136363434376130323632363833
-32363764656133653462633132366630396233613463643833353038316232353962623761373966
-35613132306662663937366464373034336531366262643939333137613031666162383434313664
-65363332383330623863326465353039306338393230303165656366386161656239316161336633
-61616434363637376131316135643336653636313565613035643038386332356263383061643262
-63333439303736623835653636353261363532646266663331313566363562343730303436353636
-63636237343965316366613137353062393832373133316438656433313132643061633130396662
-65346530313962306238373634366266386232613566656337396663626535393964323737323939
-66333165356138353362336166376466303965663439636432653862316433376264376563313137
-30653565386636363730623630646633633832306131626235613862393239386437386561356663
-64303862616530393838383364643266633336326665323961643935356339316564316332633635
-32323239333339633838666233376165303061373465376133383863333232663065626462303362
-37613638303266356537633965323966363331383130613963363933656531663031643030396632
-38343838383731346536616364303531373030643730626165633464646137376439643637633636
-62346664633931376438313639613165306363356532346566336336363662353233613134346132
-30646633363364363234396435323037306663633863383530383034383338636639326362626639
-64313532313939316562353561323534313438346530303536333066393863613136646664353264
-65663530366238343732396430633138613664653962366338643738623636373031363839393534
-38393439393366396262643461303361613666336631313037306231363537346364346432616539
-64616431633730333233616362363638613837356638343131376262363736666532356532653330
-64353431663261623033323337373932396436656633396439323162626563653630633239306233
-37633939363536396439643663623434316161383938393864653437316535313433336461383065
-32396536313835393030313136356530623539646361633935333362373339393662393335353361
-37353366666264613563663836666332353561666464623561653238666537316233343838663363
-38616463643935376233363031323838393163306465333030393331663231623061646530393237
-33356466363261653235336134376166343537643065663235333161383366336132316666383731
-39326662373162313861363132653531323165393939313461663262316337333762393134663030
-35313762356234396336356336393937316262646264306361363235663334333663363136383430
-35366538386335633038326563623833313031383762653266643034373838653863363835383466
-37323138316564383262356633646262623439326464633530313639313637326136656139376362
-35323866663262373938343863653665373139663565623730666366643630636566393266303261
-35666261383464393236353964623234306637643838623535313264356364366661653639303662
-37616531626435323664636538396439373064333563303637303562653763323935666138383338
-32326630303465366332656165343036336630383163643361306136333361373738306238393932
-63363331373434353563353631306361383431636239376464643239326531306130633463653665
-64636536383430613538623663363066646437666537306161613265323936663366353864323230
-32666333326630636433656166343331396166383438313866306566356238653061343461653035
-6236383630643231646264333133343738663663323332336536
+66356262646437623239383039643762363139653565356263623438303833663431396566353163
+3631323563343638346464386661346235656130383238370a353236623366313561646334363639
+35346232376138666430363437346638326264646562386436323962633630663136626462343762
+3862333139646636650a656337306533313436333239653561333231363030633339303835396637
+61633763313464353933306333383061346563623637383166646332303237653065346233623134
+39386130613866386235376266643062653833643132633235393931636366376164306331303762
+33393562303738393364323465653163633730633636343764646337383733643362396132303063
+32336461396561663063633137613234356535383863346334666465643437613962376439356433
+66353532333933343462636163353637613637356530333331636363616265376263636538656566
+62643137623933363237343865613131383766633561636664616434393538646332346132336630
+62333361626339626638333666393735373834396431343965386532313834353837336664306530
+36303138373734343735653637663466353435306166316661353738353035366535333265626330
+36396538353566316634376662383564323861303063386163663061323435656261316439336539
+65323063343961646435373134303039373139333265633838313636373463316231633832306261
+36623934343731343938306332356261646264613063393834336332666230626534653464363335
+66356331666434623735613333303466656263313964363030316231303330303735373135653763
+30323635303435656639653037366430316330333033306339366361353765656337316561366263
+38613265363565373832636239396362616435303536646465326132626239353361613137626632
+30353764323430656631366134373263343861376666346166363131333137376466393438346461
+31613435363637306432343437653636356665373838626361306332316664343266356666353238
+35303464326633326438353730393462373437313864326638323262323034346464643038386462
+34613435326432663566386331333430663336656239353131633261626637613035636662346136
+33626131626435316566393831333534393663326465633531373932626631613063666165613766
+63636463656435386333323037656533306331646561363061613537663738326162656332393161
+62313338376630643031666630663337646439623862306663303335303232306161353336643334
+30386537336265383334336431653637303836376261303038653739396263666161653365356133
+62636638393465363239616264316331316139363063313431323165313237646437643361653539
+63393932623934313062376137626266363330623766646231353561646332353565643638356633
+39656565363962653038393430363666643361343131323231663566343035643631646433306464
+64343735393937333036623139303531343338373162396635613863316466393235343865333065
+63353662313737353737353736316261346466366666666536653064356130326334326137383933
+64306331663430313838376266613763343537613965303937393364396238663266363539313531
+33616133646365396336613234636630396239316633333832653138393964616261643531663836
+36666265386134383430626637643766636439393636336436366265396239303335306563653962
+62323763626265373037343766386433646637633838643933633738333037633336646663316335
+36373534663439306137353236343565636538636333313239373936653263366637613335663233
+32306536643866656461343666396565373661353561393665643162613539633634666138626266
+35613663376465626363373864626266393335396461316338653262316131643633666361333537
+64366533633535313633306535393732383436333762616539646463366665623866323463336263
+38313365326338386664363538633232343636303135356230393065613238386366323435633931
+37353537313330396463383233616532653139376263346264623366383339306139386231636466
+35383761373733386133386436316431613265306264656630353561333736613734653164333031
+32643237366534643062653065333632343563633034663266393961353763663534353466383561
+38373332323135353932343231353433366637333437636238623235663836393735346230623438
+37616639643231396564373033636162616234366262333461343139363539316332333431636565
+64396639383462386239613035303565376163633637653137373631383864353532373465626630
+63636531653063306131623735393963623836313566643565386336636565633130383264316533
+33343030316365373164663661306161363062646361646332393935373533626331623163323038
+61316666366263313531373530366639333364336563623162323938356662626565383363303533
+61323730313161633035646435346234366661386339346663363438663862636361343033306134
+66396365653634373235623837666537323230616334663263633135316433313734663863333561
+34326134613335346661316632353764343235326630316535306666326662613436383764646366
+33616135333038333834656439383733656436663433303536313961333662366564376338326661
+35393839643663333130383965663538393931363662336165636435643132353461343266363265
+66346635313337346661646461386336386461346133323435643939616234666665616261313731
+38383265643830646364373164656133396132666236313265346337643939613831303366393530
+63333330346131656134366237323661316534323330306330616564393130353136306461666361
+62376637646230623264363437313161396130616231346133346665303834623833663062663436
+30643736376638303065356161366561353565343139343731313263393933313239363363636638
+32663932626332626366333635616335316235363561373539623265663733373030373439373831
+65333763653635613964396437333833626535643666643433393031336461366562333934666234
+62383632303436633965643430363763353361613762306361613066356136383131383232313961
+30626531353933336563656139316338616565303261363933306164393966343037353431653233
+32636631373966323838333666313964323933306638653162646166623237343662646134383337
+37383039383531656337626334386638613133343133633637343738343731326663356666333161
+61323765636135306563323064346465633537613361383761393361353363396237333438306166
+35386338356135633365396661356535623232613765653931303063353538643130323439366438
+39363936316263333562646133646464346362343761336662353764613261396364626164313664
+31383839303361323461663662393736616363646232666633323134653138306161323735386638
+30333365663064366535656134323338663566616163333634353939363533356364393563363333
+65623838323034336530326363383237373563326334343066323738663833353930663930373965
+30363766633137396561376636636566363862353062663038653534623430643766633631356562
+63623762343433626363353239376539633136333038623338303961373938396632313966666532
+33633865626462636137343436386661353561623035656435623566613530623163316465383664
+37346261616239653138623262623930316432633531313037353938396164646534316564613238
+36643137303965316238346134323639386261633338363130323134336632636564636638653861
+38346566643966386465366535393166626463313039613236393831393732613537336530393936
+31613234333836333732323165653931336565316564346239336663363736323763623962383261
+65626332626565636162336264303964323139303831656262373161303861393366353237393136
+62303361636131646335633733373966626135396630623133623733373565306234646165656138
+35383065613365363438396332306236646432663731326666613838643434393866623163316631
+33633931373135336438643735646266383462386431666330363066386538386335383231653334
+64323366363032386662333662613562333864356238626161366566356564343339343838373437
+30363133363136663036656461343534666636343932313336336637376665393564313234373430
+39356362616539366237646363646364326664653232313235653831663539353530393261346532
+38653134623732313466386635313730303639613561323665356633653836393237333737323864
+36323736326239636565346536646566656138666630396366353930383237333338626532663131
+64366339303230663530626632623066613030663533366539313866646466373439663831363936
+36366636653032343064383661383664636463343165326532653861663537633238323937363839
+65333030653566306465643035633162393534323731346330333861353236393732346130313034
+64303736323030643533633539623638643232306435663238393566376564313162633334613462
+33363931383664633433363032643536356135626662653733303339656339373735643732626535
+35316131356135393933393462336163373062653133356133393830393638313838396361336635
+62323463333464353939346233333462646161633036643332313337353065376561336630383163
+38306666666634393766366436653436623138363830343636323833353365333762343533633835
+66363730383932303661313831663566613830393530616164323963636339393339623065316534
+38373662636330336632653335316463623935653961363566396132616433633831636334306236
+34393761663962623563333534383739613965353033646235653532303330343961656639333337
+31643462663538313930383038343964326463363935396365343466643236653738626639616434
+62343337356239316162383564666131663535313633373062353030336439396533376235386131
+31653865643931323364356566316435333336316166643063613164643266313834323235353863
+34623936613366303161353165386662376465656366656561613466383830326162336238656236
+30646632396231653263373631346330376461303231373733646231373462623131303633616166
+37366435353830303234393962663334656335356661333437363637663664323430623135346262
+33303233343362336261633663336635366233313466636233333836663464633637383964313133
+31363461336237373337353438396132616134303432336533653737333538383534363735316334
+62303666363532383131383833383536333733306163366339653038313930646531653565386335
+38313561303062336232333634343866393763636131333665396430306432373734613834316636
+61306138626530616230343462613036323166656334656637616139663030623932386132393762
+36363364343737613165376365323361363233616532643866303632653063343431613338373633
+63303263626434326539316439323239616632653632346538336535373336623739323931623462
+35656365336630323035616161656237353135373366376466326264323863336166633438663834
+63646562386339363235396163646334333238323932383230643964366466323064313138323464
+62636239663635336332633334656631383261306437333963363464303430666462633333666638
+37633830663439396133323861303735346561376434386131363864316332623137376139376231
+30303765646132306261363335366565306564373261663336613336336632616231363366316135
+65303966616439643863326662656439623938373833323738646535636230663136653266653533
+63326235376633333366373436333534336436366434363461393763633437616262326133623563
+37613161303161356632316335616639373564306161623339353066383163303431313333306635
+36326435376661343237303633316462613532313232643461313065376134393338613130336161
+38326463303562666662613361656161306139653431386630303561626666613037393238313463
+62623533396532326332326636346134616236653638363837643131343166333564323538653638
+63383261383265353331663864643933373562396565346163396434383765343962336363353261
+38353034643436626632623235396261326235323937646135326532313561396361363062333366
+32343666343163636233646466613633383034653438646262666133643830333834383233343136
+33646437396137613466643031323538626332613066333065643237353263636435393964356464
+34383265303364663233326432313230336436373866303435336666343832383737303030306534
+36383036363937393965386431343561333961663565333565333261323033616630343130613934
+32383166333063623235333464343864326531333165346261643131393264346534393035393431
+64636139383232383332623866323363623666326361386134353030376366323039393033653061
+63383338303734323637386366646365663032636463336261643533663638343661613462373736
+32663566666133656165396233643334343335333731333938376236633330366262366139326133
+32653163343963333739303061643936363861313938646164316564393261663765393462613837
+31646636326330663464363034376230393739343961653130303166366233396262656634313061
+65646664373161356134616165383265346633376136376132643234623334353563323539333065
+61623265343730326164373063383937323766363466343466316364373935626138343938323261
+38323739663131346138623663393832346236356631633133313436363364663536316561623631
+32393562646666343537663039363165646337653564363065353433643031353865386361613534
+37633639303636313261613563353136346465326163623361323230306364396362373966336262
+31353565623239343564333533656236656666316638646666376138373164663138633835366162
+37633036346562663261343731366161613361656534343839303062373232636339643961666432
+61396665656638333435386338383638323664313139656331336131326563373235646539633835
+30393235316564656131303932386336366661373737393264613364663030363862656462646463
+33303766303166336132626265653431313263363131653835336634356632666230666235383262
+31653837356536666266383562633437376161653664353533656331303563626539626161653562
+64333337626430616330346233613566353330383361353234376236656336653735663838343333
+37643162376630373531396138643861326562323133303363363837306138373434306364356364
+64623037643737663138626262326630323638376336663062393930663566316632323862653135
+66363664336431373966663133353030393863313538366632643932363438613035326563323163
+63343565373862316230643166336438393464323032666334653631393761303337303861666134
+36663637346136623564613233343432383930616339626161623933323565623861653939326165
+37383436353533613736323237666433316130613464396333343965393031333138396461376632
+61373166643766376234646436653566313637326434363261333831336136306137333061323035
+66303335363035666366386364303861383231366339643666313636623733616562333539643465
+32353566326262326637393434656133313963646435333135613630313865373430366265666632
+37316539363439663131323835313761313765653564663337346462653332613430373037333064
+64323731623566326664613634626265383661346462323463303638346464643366663333336339
+65626265663230346338633764633436653339373031623438633638326137613635633239343834
+35373132376133363861373062386235353836633030323236343666323134326135336231613566
+34326263643163633935643065306165643564663831633561353061653331326566633664376432
+33353835366265346266663836326465356166626432356663366566313962663862363930336334
+65323337386231376361343933353865633330613466323337306137633762653931313331313339
+36663965353832386662666433383934303737656336383432633632646135383835336631373861
+63646164373137316662373330333465343134633466396566333334663036663039373934333832
+35343661303565646464373834646234623837363562386463383237373533316638666663613633
+30313437626231343062636437363836386635333862373362643532646335353863323936663034
+30316461373330623130353563643035303561303666393464636533626562633165393236663936
+31313061303465633332646363643866306439383636356366623565663437313365376331656666
+34333930333235306636646664326561613165633233353937666238646536323131313232626366
+36623237613635343134313366346661656433643136353433393532303165326564646331333035
+35353365633039353161663337306463303535643062366232336239656162653532306539333035
+65373735373832326337333962643330363338656134303637313337393466366263363264343631
+30656433643562393537326361323733643639396537303065623936336230346664616534323262
+36313864396364353836373538333130303731383437376137383938383633373063323936613464
+37613166323432363232313834326565663265336138396335313230303935396266646438323833
+65306666636338373239346332613464373235613137643133626261313031326462356138356632
+66363465346165306533613636383938636531656631306139666237396632646634333937623032
+33613439656637383537356364336265653931613835333734383836343930313838626564333065
+35363735313162616238643438316535356663396339316331633232393232316131396432653639
+36303335643534376536383331323562353139613130383330636334636261326462656565643138
+63326365363163383932653562363833643761653839306131343737353633313737353732633463
+62326563336339383661376665353230613366646132653364396338663337326136333263363838
+35626537666238383435613431666365303133373166386136353061373765313632343665306439
+34356139363636623639626633623030363933303539626665383733363638303832333561653130
+64393437386362333731643131663263626434666661326533386362306362653732376366626139
+65626662323232636134653334366639386164653535323435613863353332366630303636663339
+34333633373737333732363664353437613762383064316134393139393539306564396237343332
+64636339643030353365356136366264353935643830633237316532316265316635393664326533
+64343030396334326135336465336637623832363065353163316465303537643861306632336535
+31366338323263313965643565656430366566393130326333383862313239356437623232613865
+63383863316639346632323834633739356464313865376463323633393365653033376431373138
+61363035653233373036666531303331383337353839363563386333313730363664666463333735
+61383638363164336265383664376463616536383533666265383961353961303833353265366633
+32646330333336613631633830633837346332653437356536363532396438326364623465383133
+36656231343734643830383335623930343132386336666366623161666637623862363033373236
+32666332316266623230666231663162643262653231313965323137383532633663323239396139
+37656533373065623161623766333335353965373731346639613566343236363239313537636239
+34626533346138613764346161383337623265353563343066303236333534316137356636396565
+32616136306665346566616565346162613736333836623363346232373932366365663531653435
+37623539366233366539396361386163643161653731353232313034613135373039653339346637
+31623138326133366333656661393837653939343064656638616630346462363065623139323131
+65363036383433663566626336353432613535363239373264356237323032323634383662613763
+39646161363639326330346332353965366330346632303939623136303238323431346361656462
+61623838626139636163353430636461666239343337373132643334616234616339313761353331
+36643165373834373437363165303861306135383236303030626463373235393234313765343563
+37353932366430666539623230373161303935633565323366363232336264643961383835333066
+32623366663965316662383036306462616465353139336333643936363838313538363832363330
+33313861313636373831656431383434363664663734626363646365666135376331366335653064
+30386235313037363365363232316338613634643235383037393635333166346632356435373039
+66663862373939346434333130383065386135306661353066343330666563386136376562316134
+63346435643132316434363666343964303335663131383831373430653632663638396333393932
+36383664376666393761643730306336333538376637633135616166346432636464336434323031
+3966

ansible/host_vars/monerod_box_local/main.yml

@@ -29,3 +29,24 @@ monero_data_device: /dev/sdb
 monitored_services:
   - monerod
   - xmrblocks
+
+# ...but on THIS host, once a day rather than every five minutes.
+#
+# Both units are a Monero node in the middle of a multi-week initial sync on a
+# rotating disk. There is nothing a 5-minute poll reports that a daily one does
+# not, and every failing poll was another Signal message - 83 of them in six
+# hours while the explorer was wedged.
+#
+# This is safe to scope to the host because monerod-box runs ONLY Monero units.
+# Do not copy it to a box with a mixed workload: the timer is one per host and
+# iterates every unit in monitored_services, so slowing it slows all of them.
+#
+# 26h, not 24h: the window must EXCEED the push interval, or a check reporting
+# exactly on schedule races its own deadline and alerts on jitter alone. The
+# 900s randomized delay puts the real worst case at ~24h15m.
+#
+# Host liveness and disk are deliberately NOT slowed - they answer "is the box
+# alive", which is still wanted within minutes. See infra/400.
+service_health_on_calendar: "*-*-* 05:15:00"
+service_health_randomized_delay: "900"
+service_health_heartbeat: "26h"

ansible/host_vars/music_box_local/main.yml

@@ -0,0 +1,11 @@
+---
+# Navidrome's HTTP port. Read TWICE: by roles/navidrome on this box, and by the
+# Caddy play on the edge host, which reads it as
+# hostvars['music_box_local'].navidrome_port. A role default cannot reach a
+# second play, which is why this lives here and not in defaults/main.yml.
+navidrome_port: 4533
+
+# Systemd units monitored every 5 minutes by infra/401_service_monitoring.yml.
+# beets is not here: it is an interactive CLI, not a service.
+monitored_services:
+  - navidrome

ansible/infra/401_service_monitoring.yml

@@ -64,7 +64,7 @@
           'name': (item.0.host | lower | regex_replace('[/_.,# +&]', '-')) ~ '/' ~ item.1,
           'group': 'services',
           'token': gatus_push_tokens[item.0.host],
-          'heartbeat': '11m'}] }}"
+          'heartbeat': hostvars[item.0.host].service_health_heartbeat | default('11m')}] }}"
       loop: "{{ host_units | subelements('units') }}"
 
     - name: Register the service endpoints
@@ -98,7 +98,16 @@
         healthcheck_check: systemd-units
         healthcheck_units: "{{ monitored_services }}"
         healthcheck_units_key_prefix: "services_{{ host_key }}"
+        # Cadence and heartbeat window are ONE decision, and both are a property
+        # of the host - monerod-box runs a multi-week sync nobody needs polled
+        # every five minutes, so it declares a daily schedule in its host_vars
+        # and a matching window above. Everything else keeps 5min/11m.
+        #
+        # An empty healthcheck_on_calendar means "use healthcheck_interval" -
+        # see the timer template in roles/healthcheck.
         healthcheck_interval: "5min"
+        healthcheck_on_calendar: "{{ service_health_on_calendar | default('') }}"
+        healthcheck_randomized_delay: "{{ service_health_randomized_delay | default('30') }}"
         healthcheck_boot_delay: "2min"
         # The per-unit results go to keys under this collection; the role's own
         # single-result push is unused here, so only the base is set.

ansible/infra/402_public_monitoring.yml

@@ -31,8 +31,12 @@
       - {sub: "{{ subdomains.memos }}",              host: vipy}
       - {sub: "{{ subdomains.mempool }}",            host: vipy}
       - {sub: "{{ subdomains.datum_gateway }}",      host: vipy}
-      - {sub: "{{ subdomains.monero }}",             host: vipy}
-      - {sub: "{{ subdomains.monero_explorer }}",    host: vipy}
+      # Daily rather than the 6h default - see the Monero note under
+      # public_sites. A record that has not moved in a year does not need
+      # four looks a day while the node behind it is mid-sync.
+      - {sub: "{{ subdomains.monero }}",             host: vipy, interval: '24h'}
+      - {sub: "{{ subdomains.monero_explorer }}",    host: vipy, interval: '24h'}
+      - {sub: "{{ subdomains.music }}",              host: vipy}
 
     # A public resolver on purpose: this must test what the internet sees, not
     # what a local cache or the tailnet's MagicDNS happens to answer.
@@ -55,8 +59,22 @@
       # 401 is CORRECT: monerod answers an unauthenticated request with an HTTP
       # Digest challenge. Asserting 200 here would go green precisely when
       # rpc-login broke and the node started serving the world for free.
-      - {name: monero,      sub: "{{ subdomains.monero }}",             path: "/get_info", status: 401}
-      - {name: exmrplore,   sub: "{{ subdomains.monero_explorer }}",    path: "/",       status: 200}
+      #
+      # Both Monero sites are polled ONCE A DAY, not every 5 minutes. The node
+      # is a multi-week initial sync on a rotating disk and the explorer reads
+      # the same LMDB, so a hard poll measures the disk more than the service -
+      # and a wedged explorer turned that cadence into 83 Signal messages in
+      # six hours. `threshold` drops with it: 3 failures at 5m is 15 minutes of
+      # tolerance, but at 24h it would be THREE DAYS before anyone heard. A
+      # daily check has no room for a blip, so it alerts on the first one.
+      - {name: monero,      sub: "{{ subdomains.monero }}",             path: "/get_info", status: 401, interval: '24h', threshold: 1}
+      - {name: exmrplore,   sub: "{{ subdomains.monero_explorer }}",    path: "/",       status: 200, interval: '24h', threshold: 1}
+      # /app/, not "/": checked live, "/" answers 302 (Navidrome redirects to
+      # the SPA) and only /app/ returns 200 with the real UI. 200 rather than
+      # 401 because Navidrome has no basic auth in front of it - the Subsonic
+      # API the mobile clients speak cannot carry a proxy credential, so its
+      # own login is the gate. See roles/navidrome/README.md.
+      - {name: music,       sub: "{{ subdomains.music }}",              path: "/app/",   status: 200}
 
     # Ports published from the edge host by socket_proxy.
     public_tcp:
@@ -99,7 +117,7 @@
           'name': item.sub ~ '.' ~ root_domain,
           'group': 'dns',
           'url': dns_resolver,
-          'interval': '6h',
+          'interval': item.interval | default('6h'),
           'dns': {'query-type': 'A', 'query-name': item.sub ~ '.' ~ root_domain},
           'conditions': ['[DNS_RCODE] == NOERROR',
                          '[BODY] == ' ~ hostvars[item.host].ansible_host],
@@ -118,10 +136,10 @@
           'name': item.name,
           'group': 'public',
           'url': 'https://' ~ item.sub ~ '.' ~ root_domain ~ item.path,
-          'interval': '5m',
+          'interval': item.interval | default('5m'),
           'conditions': ['[STATUS] == ' ~ item.status,
                          '[CERTIFICATE_EXPIRATION] > 168h'],
-          'alerts': [{'type': 'signal', 'failure-threshold': 3,
+          'alerts': [{'type': 'signal', 'failure-threshold': item.threshold | default(3),
                       'success-threshold': 2, 'send-on-resolved': true,
                       'minimum-reminder-interval': '6h'}]}] }}"
       loop: "{{ public_sites }}"

ansible/infra/403_service_probe_registration.yml

@@ -45,8 +45,16 @@
       - {name: mempool-mariadb,  host: mempool_box_local}
       - {name: mempool-backend,  host: mempool_box_local}
       - {name: mempool-frontend, host: mempool_box_local}
-      - {name: monerod,          host: monerod_box_local}
-      - {name: xmrblocks,        host: monerod_box_local}
+      # The two Monero probes push ONCE A DAY, not every 5 minutes, so they
+      # need a window wider than a day. 26h, not 24h: the window must exceed
+      # the push interval or a check that reports exactly on time races its own
+      # deadline and alerts on jitter. The timers are OnCalendar 05:00 and
+      # 05:30 with RandomizedDelaySec=900, so the true worst case is ~24h15m.
+      # Cadence lives in roles/{monero,xmrblocks}/defaults/main.yml; the two
+      # are one decision and must move together.
+      - {name: monerod,          host: monerod_box_local, heartbeat: '26h'}
+      - {name: xmrblocks,        host: monerod_box_local, heartbeat: '26h'}
+      - {name: navidrome,        host: music_box_local}
 
   tasks:
     - name: Build the probe endpoint list
@@ -55,7 +63,7 @@
           'name': item.name,
           'group': 'probe',
           'token': gatus_push_tokens[item.host],
-          'heartbeat': '11m'}] }}"
+          'heartbeat': item.heartbeat | default('11m')}] }}"
       loop: "{{ probes }}"
 
     - name: Register the probe endpoints

ansible/inventory.ini

@@ -16,6 +16,11 @@ forgejo_runner_local ansible_host=forgejo-runner-box lan_ip=192.168.1.132 ansibl
 arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.147 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 small_backups_local ansible_host=small-backups-box lan_ip=192.168.1.131 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+# lan_ip read off nodito's ARP table for the VM's own MAC (bc:24:11:89:fc:1f),
+# not guessed. Note .149 previously belonged to another machine - its stale host
+# key was still in known_hosts and SSH refused the connection, which is the
+# designed behaviour described in group_vars/nodito_vms.yml.
+music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 
 # Local connection to laptop: this assumes you're running ansible commands from your personal laptop
 [lapy]
@@ -55,6 +60,9 @@ monerod_box_local
 [memos]
 memos_box_local
 
+[music]
+music_box_local
+
 [ci_runner]
 forgejo_runner_local
 

ansible/roles/monero/defaults/main.yml

@@ -94,3 +94,16 @@ healthcheck_push_url: ""
 # Bearer token for the Gatus external endpoint. Required whenever a push URL
 # is set: Gatus rejects an unauthenticated push with 401.
 healthcheck_push_token: ""
+
+# WHEN TO CHECK. Daily, at a fixed hour, rather than an OnUnitActiveSec
+# interval that drifts with every restart.
+#
+# Prefixed because the generic `healthcheck` role owns the unprefixed
+# healthcheck_on_calendar; an unprefixed default here would shadow it for any
+# play that includes both.
+#
+# Paired with the 26h Gatus heartbeat in
+# infra/403_service_probe_registration.yml. Change one and you must change the
+# other: the window has to stay above the interval or the check alerts on its
+# own jitter.
+monero_healthcheck_on_calendar: "*-*-* 05:00:00"

ansible/roles/monero/templates/healthcheck.timer.j2

@@ -8,7 +8,17 @@ Description=Monero Node Health Check Timer
 
 [Timer]
 OnBootSec=2min
-OnUnitActiveSec=5min
+# ONCE A DAY, not every 5 minutes.
+#
+# A 5-minute poll of monerod buys nothing a daily one does not: the node is a
+# multi-week initial sync on a rotating disk, and every failing poll is another
+# Signal message. The cadence and the Gatus heartbeat window are ONE decision -
+# the window MUST exceed this interval, and it is set to 26h in
+# infra/403_service_probe_registration.yml. A 24h push under a 24h window races
+# itself and alerts on jitter alone.
+OnCalendar={{ monero_healthcheck_on_calendar }}
+# Spread against the other daily pushes leaving this box.
+RandomizedDelaySec=900
 Persistent=true
 
 [Install]

ansible/roles/navidrome/README.md

@@ -0,0 +1,179 @@
+# navidrome
+
+Navidrome — a music server — plus [beets](https://beets.io), the CLI that fixes
+tags before Navidrome ever indexes them. Both live on `music-box`.
+
+Navidrome is a single Go binary with a SQLite database and no external
+dependencies beyond `ffmpeg` for transcoding. It speaks two protocols on one
+port: its own web UI, and the **Subsonic API** (`/rest/*`), which is what every
+mobile client in the ecosystem — Symfonium, Amperfy, Tempo, substreamer —
+actually talks to.
+
+| | |
+|---|---|
+| Host | `music-box` (nodito VM 8200, 1 core / 1 GB / 50 GB) |
+| Port | 4533 |
+| Public at | `music.contrapeso.xyz`, proxied by Caddy on `vipy` |
+| Library | `/srv/music` — beets-managed, Navidrome reads only |
+| Staging | `/srv/incoming` — unsorted, Navidrome never looks here |
+| State | `/var/lib/navidrome` — SQLite, cache, transcode scratch |
+| Backups | none, deliberately — see below |
+
+## Exposure
+
+Public, authenticated by **Navidrome's own login**. There is no `basic_auth` in
+front of it, and that is a decision rather than an omission.
+
+The reason is the Subsonic API. A Subsonic client authenticates with
+`?u=<user>&t=<md5(password+salt)>&s=<salt>` in the query string and has nowhere
+to put a second set of credentials. HTTP Basic Auth in front of the whole site
+is invisible to most of them, so the choice is between a proxy credential and a
+working phone. This estate has made the same call before, for the same reason:
+`monerod`'s restricted RPC is proxied bare so Feather can reach it, and its own
+Digest auth does the authenticating.
+
+The cost is stated plainly: Navidrome's login page and the whole `/rest/*`
+surface are reachable by anyone who finds the hostname. Navidrome has **no rate
+limiting and no 2FA**. The fail2ban jail below covers part of that gap.
+
+### The fail2ban jail on `vipy`, and what it does not cover
+
+Installed by the edge play in `services/navidrome/deploy_navidrome_playbook.yml`,
+reading a per-vhost Caddy access log at `/var/log/caddy/music.log`.
+
+It catches **web UI brute force**: a bad login on `/auth/login` returns HTTP
+401, which lands in the access log and trips the filter.
+
+It very likely does **not** catch Subsonic API brute force, because the
+Subsonic protocol answers a bad password with **HTTP 200** and puts the failure
+in the body (`<subsonic-response status="failed"><error code="40" .../>`).
+Caddy never sees response bodies, so a status-code filter is blind to it.
+
+The obvious fix does not work either. Navidrome's own journal logs those
+failures, but the request arrives from `vipy` over the tailnet, so the only
+address fail2ban could ban on this box is `vipy`'s — which would take down every
+service behind the edge. Banning has to happen at the edge; the edge cannot see
+the failure.
+
+So: `/rest/*` is protected by password strength alone. That is the reason the
+Navidrome password must be long, random, and used nowhere else.
+
+To re-check whether this is still true after a Navidrome upgrade:
+
+```bash
+curl -s -o /dev/null -w '%{http_code}\n' \
+  'https://music.contrapeso.xyz/rest/ping.view?u=x&p=wrong&v=1.16.1&c=test'
+```
+
+`200` means the caveat holds. `401` means it no longer does — widen the jail
+and delete this section.
+
+## Passwords are stored reversibly
+
+Navidrome **cannot** hash user passwords one-way. The Subsonic API requires the
+server to compute `md5(password + salt)` on every request, so it must be able to
+recover the plaintext. It encrypts passwords at rest with
+`PasswordEncryptionKey` instead (`navidrome_password_encryption_key`, from
+`group_vars/all/vault.yml`).
+
+Two consequences, both real:
+
+- Root on this box means every Navidrome password in plaintext. **Use a password
+  unique to this service.**
+- **Changing the key invalidates every stored password.** Set it before the first
+  account exists and leave it alone.
+
+## Backups: none, deliberately
+
+Recorded as a decision so it does not read as an oversight — the same treatment
+`phoenixd`, `monero` and `xmrblocks` get.
+
+The music files are large and would swamp the age-encrypted nightly path that
+`backup_source` uses. `navidrome.db` is mostly a rebuildable index: delete it and
+a rescan reconstructs the library.
+
+What is **not** rebuildable is the curation on top — playlists, play counts,
+ratings, starred items, user accounts. Today there is none of it, which is why
+this is a fair trade. **If that changes, this is the decision to revisit**, and
+the fix is small: `navidrome.db` is a single SQLite file and would fit the
+`backup_source` pattern in about fifteen lines.
+
+The beets database (`/var/lib/beets/library.db`) is in the same position — it
+can be rebuilt from the tags beets itself wrote into the files.
+
+## beets
+
+Installed and configured by this role; **not** a service, and deliberately so.
+`beet import` is interactive and it *moves and renames files*. A timer that
+reorganises a music library unattended is a way to lose one.
+
+Config at `~counterweight/.config/beets/config.yaml`, library DB in
+`/var/lib/beets`.
+
+```bash
+# the usual loop
+rsync -avP ~/Music/newstuff/ music-box:/srv/incoming/
+ssh music-box
+beet import /srv/incoming
+```
+
+Navidrome picks up the result on its next scan.
+
+The single most important line in that config is
+`paths.default: $albumartist/...`. Organising by `$artist` instead scatters
+every compilation and every "feat." track into a phantom artist, which is how a
+library ends up unusable and Navidrome gets blamed for it.
+
+ReplayGain and lyrics are set `auto: no` — on one core they turn a two-minute
+import into a twenty-minute one. Run `beet replaygain` and `beet lyrics`
+afterwards, separately.
+
+### Ownership, which is the fiddly part
+
+`/srv/music` is `counterweight:navidrome` mode **2750**. The setgid bit is
+load-bearing: without it, files beets creates land in the curator's own group,
+Navidrome cannot read them, and the symptom is an **empty library rather than a
+permission error**.
+
+## Monitoring
+
+| Layer | What it answers | Where |
+|---|---|---|
+| `infra/400` | is the host alive, is the disk filling | automatic, needs `gatus_push_tokens.music_box_local` in the vault |
+| `infra/401` | is the `navidrome` unit active | `monitored_services` in `host_vars/music_box_local/main.yml` |
+| `infra/403` + this role | does Navidrome actually serve its own UI (`/app/`) | `probe_navidrome` |
+| `infra/402` | does `music.contrapeso.xyz` answer from outside, is the cert valid | `public_sites` |
+
+The role's own check fetches `/app/` and greps for `navidrome`. A unit can be
+perfectly `active` while serving nothing useful.
+
+`/app/` was chosen after testing all three candidates against 0.64.0:
+`/ping` returns a bare `.` (proves only that something listens);
+`/rest/ping.view` identifies the server but logs two warnings per call,
+~576 a day; `/app/` returns `<title>Navidrome</title>` and logs nothing.
+
+## Upgrading
+
+```bash
+curl -sL https://github.com/navidrome/navidrome/releases/download/v<ver>/navidrome_checksums.txt \
+  | grep linux_amd64.tar.gz
+```
+
+Bump `navidrome_version` and `navidrome_sha256` **together** in
+`defaults/main.yml`, then re-run the playbook. The install is version-gated, so
+an unchanged version is a no-op.
+
+## Sizing
+
+1 core / 1 GB / 50 GB. Two things to know:
+
+- **50 GB is the library ceiling**, not just the OS — `/srv/music` is on the root
+  disk. Roughly 300–450 albums in FLAC. Growing it means a Proxmox resize plus
+  `growpart`/`resize2fs` (see `03_vm_disk_enlargement.md`), because
+  `ignore_changes = all` in the tofu config means tofu will not do it.
+- **1 GB is fine for serving, tight for scanning.** The initial scan of a large
+  library is the memory peak. If it gets OOM-killed, do what `monerod-box`
+  needed: raise it in the Proxmox UI and record it back in `vms.auto.tfvars`.
+
+Transcoding is the CPU cost, and it is avoidable: clients that direct-play
+(Symfonium, Amperfy) make the server a file shipper. Roku and browsers transcode.

ansible/roles/navidrome/defaults/main.yml

@@ -0,0 +1,85 @@
+# Navidrome Configuration Variables
+# https://www.navidrome.org/docs/usage/configuration-options/
+
+# --- Version ----------------------------------------------------------------
+# Bump navidrome_version and navidrome_sha256 TOGETHER. The hash is the one
+# for the linux_amd64 tarball in that release's navidrome_checksums.txt:
+#   curl -sL https://github.com/navidrome/navidrome/releases/download/v<ver>/navidrome_checksums.txt \
+#     | grep linux_amd64.tar.gz
+navidrome_version: "0.64.0"
+navidrome_arch: "linux_amd64"
+navidrome_sha256: "efd94d11253234035b5d3cf284da917111736e72daeeda4f45a58251c60553ac"
+navidrome_url: "https://github.com/navidrome/navidrome/releases/download/v{{ navidrome_version }}/navidrome_{{ navidrome_version }}_{{ navidrome_arch }}.tar.gz"
+
+# --- Service user -----------------------------------------------------------
+navidrome_user: navidrome
+navidrome_group: navidrome
+
+# --- Paths ------------------------------------------------------------------
+navidrome_bin_path: /usr/local/bin/navidrome
+navidrome_config_dir: /etc/navidrome
+navidrome_config_file: "{{ navidrome_config_dir }}/navidrome.toml"
+# SQLite database, cache, transcoding scratch. The only thing Navidrome writes.
+navidrome_data_dir: /var/lib/navidrome
+
+# The canonical library. Navidrome READS this and never writes it, which is why
+# it is absent from ReadWritePaths in the unit. beets is what writes here.
+navidrome_music_dir: /srv/music
+# Staging area. Whatever lands here is unsorted and has NOT been through beets;
+# Navidrome never looks at it.
+navidrome_incoming_dir: /srv/incoming
+
+# The human who curates the library with beets. Owns both directories above;
+# navidrome only needs group read.
+navidrome_curator_user: "{{ new_user }}"
+
+# --- HTTP -------------------------------------------------------------------
+# 0.0.0.0, not loopback: the edge host reaches this across the tailnet. UFW
+# denies the port from the LAN and the internet, and tailnet traffic bypasses
+# UFW entirely (-A INPUT -j ts-input sits ahead of every ufw-* chain), so
+# binding wide here does not widen exposure. Confirmed on mempool-box.
+navidrome_bind_address: "0.0.0.0"
+navidrome_port: 4533
+navidrome_log_level: info
+
+# Navidrome's telemetry. Off.
+navidrome_enable_insights: false
+
+# --- Secrets ----------------------------------------------------------------
+# Navidrome CANNOT store user passwords as a one-way hash: the Subsonic API
+# authenticates with md5(password + salt), so the server has to be able to
+# recover the plaintext. It encrypts them at rest with this key instead.
+#
+# Consequences, both real:
+#   - root on this box means every Navidrome password in plaintext. Use a
+#     password unique to this service and nowhere else.
+#   - CHANGING THIS KEY INVALIDATES EVERY STORED PASSWORD. Set it before the
+#     first account exists and then leave it alone.
+# Injected by the playbook from group_vars/all/vault.yml; empty here so the
+# role stays self-contained and asserts loudly rather than starting insecure.
+navidrome_password_encryption_key: ""
+
+# Set true to see the rendered config in --diff. It carries the key above, so
+# the template is diff-suppressed by default.
+navidrome_reveal_config: false
+
+# --- beets ------------------------------------------------------------------
+# beets is NOT a service. It is an interactive CLI the curator runs over SSH to
+# fix tags before Navidrome ever indexes them; there is no unit and no timer.
+navidrome_beets_enabled: true
+navidrome_beets_dir: /var/lib/beets
+navidrome_beets_config: "{{ '/home/' ~ navidrome_curator_user ~ '/.config/beets/config.yaml' }}"
+
+# --- Health check -----------------------------------------------------------
+# "Does Navidrome actually answer", as opposed to infra/401's "is the unit
+# active". The exit code is the answer and systemd keeps it:
+#     systemctl is-failed navidrome-healthcheck.service
+navidrome_healthcheck_script_path: /usr/local/bin/navidrome-healthcheck-push.sh
+navidrome_healthcheck_service_name: navidrome-healthcheck
+
+# WHERE TO REPORT HEALTH — the one place to plug in monitoring.
+# Empty means check, log, exit honestly, report nowhere.
+healthcheck_push_url: ""
+# Bearer token for the Gatus external endpoint. Required whenever a push URL
+# is set: Gatus rejects an unauthenticated push with 401.
+healthcheck_push_token: ""

ansible/roles/navidrome/handlers/main.yml

@@ -0,0 +1,16 @@
+---
+# NO `when:` on either of these, ever. The uptime_kuma_enabled banner was once
+# applied to contiguous blocks and took Fulcrum's entire restart handler with
+# it: config changes landed on disk and silently never took effect for months.
+# A handler that can be gated off is a handler that will be.
+- name: Restart navidrome
+  systemd:
+    name: navidrome
+    state: restarted
+    daemon_reload: yes
+
+- name: Restart navidrome health check timer
+  systemd:
+    name: navidrome-healthcheck.timer
+    state: restarted
+    daemon_reload: yes

ansible/roles/navidrome/tasks/beets.yml

@@ -0,0 +1,70 @@
+---
+# beets: the tag fixer that runs BEFORE Navidrome sees anything.
+#
+# Navidrome has no metadata fetcher and no tag editor by design - it trusts the
+# tags on disk completely. That is the right call, but it means a library with
+# bad tags renders as a bad library and there is nothing in Navidrome to fix it
+# with. beets is the fix, and it belongs on this box because that is where the
+# files are.
+#
+# There is no unit and no timer here on purpose: `beet import` is interactive,
+# it MOVES and RENAMES files, and a cron job that reorganises a music library
+# unattended is a way to lose one.
+
+- name: Install beets and its fingerprinting dependencies
+  apt:
+    name:
+      - beets
+      # fpcalc. Without it the chroma plugin cannot acoustically identify a
+      # file whose tags are too mangled to match, which is exactly the case
+      # beets is most useful for.
+      - libchromaprint-tools
+      - python3-acoustid
+      # lastgenre imports pylast, and Debian's beets package does NOT depend on
+      # it. Without this the plugin fails to load and every single `beet`
+      # invocation prints a ModuleNotFoundError traceback while genres are
+      # silently never fetched - configured, listed in `plugins:`, and inert.
+      - python3-pylast
+    state: present
+    update_cache: true
+  when: navidrome_beets_enabled | bool
+
+- name: Create the beets state directory
+  ansible.builtin.file:
+    path: "{{ navidrome_beets_dir }}"
+    state: directory
+    owner: "{{ navidrome_curator_user }}"
+    group: "{{ navidrome_curator_user }}"
+    mode: '0750'
+  when: navidrome_beets_enabled | bool
+
+# Both levels explicitly, parent first. The file module creates missing parents
+# like `mkdir -p`, but it creates them owned by ROOT - which would leave
+# ~/.config root-owned on a fresh cloud-init box and break every other
+# user-level tool that wants to write there later.
+- name: Create the curator's XDG config directory
+  ansible.builtin.file:
+    path: "{{ navidrome_beets_config | dirname | dirname }}"
+    state: directory
+    owner: "{{ navidrome_curator_user }}"
+    group: "{{ navidrome_curator_user }}"
+    mode: '0755'
+  when: navidrome_beets_enabled | bool
+
+- name: Create the beets config directory
+  ansible.builtin.file:
+    path: "{{ navidrome_beets_config | dirname }}"
+    state: directory
+    owner: "{{ navidrome_curator_user }}"
+    group: "{{ navidrome_curator_user }}"
+    mode: '0755'
+  when: navidrome_beets_enabled | bool
+
+- name: Write the beets configuration
+  ansible.builtin.template:
+    src: beets-config.yaml.j2
+    dest: "{{ navidrome_beets_config }}"
+    owner: "{{ navidrome_curator_user }}"
+    group: "{{ navidrome_curator_user }}"
+    mode: '0644'
+  when: navidrome_beets_enabled | bool

ansible/roles/navidrome/tasks/healthcheck.yml

@@ -0,0 +1,48 @@
+---
+# "Does Navidrome actually answer", as opposed to infra/401's "is the unit
+# active". Navidrome can be perfectly `active` and serving 500s - a music
+# folder it cannot read does exactly that, and presents as an empty library.
+
+- name: Create the navidrome health check script
+  ansible.builtin.template:
+    src: healthcheck.sh.j2
+    dest: "{{ navidrome_healthcheck_script_path }}"
+    owner: root
+    group: root
+    mode: '0755'
+    validate: "bash -n %s"
+
+# 0600: carries the Gatus bearer token.
+- name: Create the navidrome health check systemd service
+  ansible.builtin.template:
+    src: healthcheck.service.j2
+    dest: "/etc/systemd/system/{{ navidrome_healthcheck_service_name }}.service"
+    owner: root
+    group: root
+    mode: '0600'
+  notify: Restart navidrome health check timer
+
+- name: Create the navidrome health check systemd timer
+  ansible.builtin.template:
+    src: healthcheck.timer.j2
+    dest: "/etc/systemd/system/{{ navidrome_healthcheck_service_name }}.timer"
+    owner: root
+    group: root
+    mode: '0644'
+  notify: Restart navidrome health check timer
+
+- name: Reload systemd daemon after health check units
+  systemd:
+    daemon_reload: yes
+
+# Ungated: enabling a timer is deployment, not monitoring.
+- name: Enable and restart the navidrome health check timer
+  systemd:
+    name: "{{ navidrome_healthcheck_service_name }}.timer"
+    enabled: yes
+    state: restarted
+    daemon_reload: yes
+
+- name: Run the navidrome health check once to arm the timer
+  ansible.builtin.command: "systemctl start {{ navidrome_healthcheck_service_name }}.service"
+  changed_when: false

ansible/roles/navidrome/tasks/install.yml

@@ -0,0 +1,134 @@
+---
+# The user, the directories, and the binary.
+
+- name: Install Navidrome's runtime dependencies
+  apt:
+    name:
+      # Navidrome shells out to ffmpeg for on-the-fly transcoding. Without it
+      # the server starts fine and every format the client cannot play natively
+      # simply fails, which looks like a client bug.
+      - ffmpeg
+      - curl
+    state: present
+    update_cache: true
+
+- name: Create the navidrome system group
+  group:
+    name: "{{ navidrome_group }}"
+    system: yes
+    state: present
+
+- name: Create the navidrome system user
+  user:
+    name: "{{ navidrome_user }}"
+    group: "{{ navidrome_group }}"
+    system: yes
+    shell: /usr/sbin/nologin
+    home: "{{ navidrome_data_dir }}"
+    create_home: no
+    comment: "Navidrome Music Server"
+    state: present
+
+- name: Create the navidrome data directory
+  ansible.builtin.file:
+    path: "{{ navidrome_data_dir }}"
+    state: directory
+    owner: "{{ navidrome_user }}"
+    group: "{{ navidrome_group }}"
+    mode: '0750'
+
+- name: Create the navidrome config directory
+  ansible.builtin.file:
+    path: "{{ navidrome_config_dir }}"
+    state: directory
+    owner: root
+    group: "{{ navidrome_group }}"
+    mode: '0750'
+
+# Two principals share this tree and neither is root: the curator writes it
+# through beets, Navidrome only reads it. setgid (the 2) is load-bearing -
+# without it, files beets creates land in the curator's own group and Navidrome
+# cannot read them, which presents as an empty library rather than an error.
+- name: Create the music library directory
+  ansible.builtin.file:
+    path: "{{ navidrome_music_dir }}"
+    state: directory
+    owner: "{{ navidrome_curator_user }}"
+    group: "{{ navidrome_group }}"
+    mode: '2750'
+
+- name: Create the incoming staging directory
+  ansible.builtin.file:
+    path: "{{ navidrome_incoming_dir }}"
+    state: directory
+    owner: "{{ navidrome_curator_user }}"
+    group: "{{ navidrome_curator_user }}"
+    mode: '0755'
+
+# check_mode: no on the next two - this repo verifies with `--check --diff`, so
+# the role has to survive a dry run. A read-only version probe and a temp
+# directory are both harmless to perform for real, and without them every task
+# below evaluates against an undefined navidrome_tmp.path and the check aborts.
+- name: Check the installed navidrome version
+  ansible.builtin.command: "{{ navidrome_bin_path }} --version"
+  register: navidrome_installed_version
+  changed_when: false
+  failed_when: false
+  check_mode: no
+
+- name: Decide whether navidrome needs installing
+  ansible.builtin.set_fact:
+    navidrome_needs_install: "{{ navidrome_version not in (navidrome_installed_version.stdout | default('')) }}"
+
+- name: Create a temporary directory for the navidrome download
+  ansible.builtin.tempfile:
+    state: directory
+    suffix: navidrome
+  register: navidrome_tmp
+  when: navidrome_needs_install | bool
+  check_mode: no
+
+# checksum: is the verification. Upstream publishes navidrome_checksums.txt per
+# release but signs nothing, so there is no signature to check the way
+# roles/monero checks binaryfate's - this pins the exact artefact and no more.
+- name: Download the navidrome release tarball
+  ansible.builtin.get_url:
+    url: "{{ navidrome_url }}"
+    dest: "{{ navidrome_tmp.path }}/navidrome.tar.gz"
+    checksum: "sha256:{{ navidrome_sha256 }}"
+    mode: '0644'
+  when: navidrome_needs_install | bool and not ansible_check_mode
+
+- name: Unpack the navidrome tarball
+  ansible.builtin.unarchive:
+    src: "{{ navidrome_tmp.path }}/navidrome.tar.gz"
+    dest: "{{ navidrome_tmp.path }}"
+    remote_src: yes
+  when: navidrome_needs_install | bool and not ansible_check_mode
+
+- name: Install the navidrome binary
+  ansible.builtin.copy:
+    src: "{{ navidrome_tmp.path }}/navidrome"
+    dest: "{{ navidrome_bin_path }}"
+    remote_src: yes
+    owner: root
+    group: root
+    mode: '0755'
+  when: navidrome_needs_install | bool and not ansible_check_mode
+  notify: Restart navidrome
+
+- name: Remove the navidrome download directory
+  ansible.builtin.file:
+    path: "{{ navidrome_tmp.path }}"
+    state: absent
+  when: navidrome_needs_install | bool
+  check_mode: no
+
+# Check mode skips the three tasks above, so without this a dry run is silent
+# about the single most important thing it would do.
+- name: Report what a real run would install
+  ansible.builtin.debug:
+    msg: >-
+      Would install navidrome {{ navidrome_version }} to {{ navidrome_bin_path }}
+      (currently: {{ navidrome_installed_version.stdout | default('not installed') | trim }})
+  when: navidrome_needs_install | bool and ansible_check_mode

ansible/roles/navidrome/tasks/main.yml

@@ -0,0 +1,7 @@
+---
+# import_tasks, not include_tasks: static imports stay visible to --list-tasks,
+# which is the main verification tool in this repo.
+- ansible.builtin.import_tasks: install.yml
+- ansible.builtin.import_tasks: beets.yml
+- ansible.builtin.import_tasks: service.yml
+- ansible.builtin.import_tasks: healthcheck.yml

ansible/roles/navidrome/tasks/service.yml

@@ -0,0 +1,67 @@
+---
+# Config, unit, start, prove it is up.
+
+# The key encrypts every stored password. Starting without it would come up
+# fine and quietly use a built-in default, so this fails the run instead.
+- name: Assert the password encryption key arrived from the vault
+  ansible.builtin.assert:
+    that:
+      - navidrome_password_encryption_key | length > 0
+    fail_msg: >-
+      navidrome_password_encryption_key is empty. It is injected by
+      services/navidrome/deploy_navidrome_playbook.yml from
+      group_vars/all/vault.yml. Navidrome stores user passwords REVERSIBLY
+      (the Subsonic API needs the plaintext to compute md5(password+salt)),
+      and this key is what encrypts them at rest. Set it before the first
+      account exists - changing it later invalidates every stored password.
+    quiet: yes
+
+- name: Write the navidrome configuration
+  ansible.builtin.template:
+    src: navidrome.toml.j2
+    dest: "{{ navidrome_config_file }}"
+    owner: root
+    group: "{{ navidrome_group }}"
+    mode: '0640'
+  diff: "{{ navidrome_reveal_config | default(false) | bool }}"
+  notify: Restart navidrome
+
+- name: Create the navidrome systemd unit
+  ansible.builtin.template:
+    src: navidrome.service.j2
+    dest: /etc/systemd/system/navidrome.service
+    owner: root
+    group: root
+    mode: '0644'
+  notify: Restart navidrome
+
+- name: Reload systemd daemon
+  systemd:
+    daemon_reload: yes
+
+- name: Enable and start navidrome
+  systemd:
+    name: navidrome
+    enabled: yes
+    state: started
+
+- name: Flush handlers so navidrome is running before we probe it
+  meta: flush_handlers
+
+- name: Wait for the navidrome HTTP port
+  ansible.builtin.wait_for:
+    port: "{{ navidrome_port }}"
+    host: 127.0.0.1
+    delay: 5
+    timeout: 300
+  ignore_errors: yes
+
+- name: Check the navidrome service status
+  systemd:
+    name: navidrome
+  register: navidrome_status
+  changed_when: false
+
+- name: Display the navidrome service status
+  ansible.builtin.debug:
+    msg: "navidrome is {{ 'running' if navidrome_status.status.ActiveState == 'active' else 'NOT running' }}"

ansible/roles/navidrome/templates/beets-config.yaml.j2

@@ -0,0 +1,81 @@
+# beets configuration — managed by Ansible (roles/navidrome)
+# https://beets.readthedocs.io/
+#
+# Run it by hand, over SSH, when material has accumulated in {{ navidrome_incoming_dir }}:
+#     beet import {{ navidrome_incoming_dir }}
+#
+# It is interactive on purpose. `beet import -q` auto-applies only strong
+# matches and leaves the rest where they are, which makes a usable review queue
+# if the batches ever get boring.
+
+directory: {{ navidrome_music_dir }}
+library: {{ navidrome_beets_dir }}/library.db
+
+# $albumartist, NOT $artist. This is the single most important line in the
+# file: organising by $artist scatters every compilation and every "feat."
+# track into its own phantom artist, which is how a library ends up with four
+# hundred one-album artists and Navidrome gets blamed for it.
+paths:
+    default: $albumartist/$album%aunique{}/$track $title
+    singleton: Non-Album/$artist/$title
+    comp: Compilations/$album%aunique{}/$track $title
+    albumtype:soundtrack: Soundtracks/$album/$track $title
+
+import:
+    # MOVES files out of {{ navidrome_incoming_dir }} into the library. Set
+    # `copy: yes` and `move: no` instead if you would rather keep originals
+    # until you trust it.
+    move: yes
+    write: yes
+    languages: en es
+    log: {{ navidrome_beets_dir }}/import.log
+
+plugins: chroma fetchart embedart scrub duplicates mbsync missing info lastgenre replaygain lyrics permissions
+
+# THE trap this library is exposed to, and the fix for it.
+#
+# beets MOVES files into place, and rename(2) preserves the original group and
+# mode - so the setgid bit on {{ navidrome_music_dir }} does NOT apply to them
+# (it only affects files CREATED there, such as the album directories beets
+# makes, which do come out group-{{ navidrome_group }}). A track that arrives
+# mode 0600 therefore lands mode 0600 owned by the curator's own group, and
+# Navidrome cannot read it. It does not error: the track is simply ABSENT from
+# the library, which is indistinguishable from never having imported it.
+#
+# Measured on this box: a 0600 file was unreadable by the navidrome user; the
+# same file at 0644 was fine. So normalise every imported file.
+# NO leading zeros. The plugin does int(str(value), 8), and YAML would parse
+# `0644` as octal -> decimal 420, which the plugin then re-reads as octal 0o420.
+# Verified on the box: `0644` silently left a 0600 file at 0600; `644` fixes it.
+permissions:
+    file: 644
+    dir: 755
+
+# Acoustic fingerprinting: identifies a file from the audio itself when the
+# tags are too mangled to match. Needs fpcalc, installed by the role.
+chroma:
+    auto: yes
+
+fetchart:
+    auto: yes
+embedart:
+    auto: yes
+# Strip junk and stale tags before writing clean ones.
+scrub:
+    auto: yes
+
+lastgenre:
+    auto: yes
+    source: album
+
+# auto: no on both of these deliberately. This box has one core; ReplayGain
+# analysis and lyrics fetching during an import turn a two-minute job into a
+# twenty-minute one. Run them separately, after the import, when nothing is
+# waiting on you:
+#     beet replaygain
+#     beet lyrics
+replaygain:
+    auto: no
+    backend: ffmpeg
+lyrics:
+    auto: no

ansible/roles/navidrome/templates/healthcheck.service.j2

@@ -0,0 +1,18 @@
+[Unit]
+Description=Navidrome Health Check
+After=network.target navidrome.service
+
+[Service]
+Type=oneshot
+User=root
+ExecStart={{ navidrome_healthcheck_script_path }}
+# Comfortably longer than the script's 30s curl plus its push retries.
+TimeoutStartSec=120
+Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
+Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }}
+Environment=NAVIDROME_URL=http://127.0.0.1:{{ navidrome_port }}
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/navidrome/templates/healthcheck.sh.j2

@@ -0,0 +1,65 @@
+#!/bin/bash
+# Navidrome health check — managed by Ansible (roles/navidrome)
+#
+# The exit code is the answer and systemd keeps it:
+#     systemctl is-failed {{ navidrome_healthcheck_service_name }}.service
+# Reporting anywhere else is optional and generic.
+#
+# /app/ rather than /ping, established by testing against 0.64.0 on the box:
+#
+#   /ping            200, Content-Type text/plain, body is a single "."  - it
+#                    proves something is listening and nothing more.
+#   /rest/ping.view  identifies itself ("type":"navidrome") but logs TWO
+#                    warning lines per call. At one call per 5 minutes that is
+#                    ~576 spurious warnings a day, which is how you train
+#                    yourself to ignore the warning level.
+#   /app/            200, 2.8 KB, <title>Navidrome</title>, and SILENT.
+#
+# So /app/ is the only one that proves it is really Navidrome serving its real
+# UI without polluting the journal. Same reasoning as the xmrblocks check,
+# which fetches a real page because a missing templates dir serves 200s of
+# nothing.
+
+URL="${NAVIDROME_URL:-http://127.0.0.1:{{ navidrome_port }}}"
+PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
+PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}"
+
+report() {
+    local status=$1
+    local msg=$2
+
+    # No push URL is normal, not an error: the exit code below is still a
+    # complete answer for anything reading unit state.
+    [ -n "$PUSH_URL" ] || return 0
+
+    local encoded_msg="${msg// /%20}"
+    local _ok=false
+    [ "${status}" = "up" ] && _ok=true
+    curl -s --max-time 15 --retry 2 -o /dev/null -X POST \
+        -H "Authorization: Bearer ${PUSH_TOKEN}" \
+        "${PUSH_URL}?success=${_ok}&error=${encoded_msg}" || true
+}
+
+# 30s: Navidrome answers /ping in milliseconds normally, but this box has one
+# core and a library scan will happily eat it. A tight timeout here would
+# manufacture false DOWNs every time a scan runs.
+body=$(curl -s --max-time 30 -w '\n%{http_code}' "${URL}/app/" 2>/dev/null)
+code=$(printf '%s' "$body" | tail -1)
+
+if [ -z "$code" ] || [ "$code" = "000" ]; then
+    report "down" "navidrome not responding"
+    exit 1
+fi
+
+if [ "$code" != "200" ]; then
+    report "down" "navidrome HTTP ${code}"
+    exit 1
+fi
+
+if ! printf '%s' "$body" | grep -qi "navidrome"; then
+    report "down" "navidrome served 200 but not its own UI (something else on the port?)"
+    exit 1
+fi
+
+report "up" "OK"
+exit 0

ansible/roles/navidrome/templates/healthcheck.timer.j2

@@ -0,0 +1,14 @@
+[Unit]
+Description=Navidrome Health Check Timer
+# Deliberately NO `Requires=navidrome.service`. Requires on a timer stops the
+# timer when the watched unit stops - "if the thing I am watching goes down,
+# stop watching it" - leaving nothing to re-arm when it returns. Fulcrum's
+# timer carried it and sat active+enabled for seven months without firing.
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=5min
+Persistent=true
+
+[Install]
+WantedBy=timers.target

ansible/roles/navidrome/templates/navidrome.service.j2

@@ -0,0 +1,33 @@
+[Unit]
+Description=Navidrome Music Server
+Documentation=https://www.navidrome.org/docs/
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User={{ navidrome_user }}
+Group={{ navidrome_group }}
+WorkingDirectory={{ navidrome_data_dir }}
+ExecStart={{ navidrome_bin_path }} --configfile {{ navidrome_config_file }}
+Restart=always
+RestartSec=20
+TimeoutStopSec=60
+StandardOutput=journal
+StandardError=journal
+
+# Hardening: Navidrome writes its database and cache, and nothing else.
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=read-only
+# {{ navidrome_music_dir }} is deliberately absent: ProtectSystem=strict leaves
+# it read-only, which is exactly right. Navidrome indexes the library; beets
+# writes it. If Navidrome ever appears to need write access here, something is
+# wrong with the request, not with this line.
+ReadWritePaths={{ navidrome_data_dir }}
+
+LimitNOFILE=65535
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/navidrome/templates/navidrome.toml.j2

@@ -0,0 +1,24 @@
+# Navidrome configuration — managed by Ansible (roles/navidrome)
+# https://www.navidrome.org/docs/usage/configuration-options/
+#
+# Mode 0640 root:{{ navidrome_group }}: PasswordEncryptionKey is below.
+
+MusicFolder = "{{ navidrome_music_dir }}"
+DataFolder  = "{{ navidrome_data_dir }}"
+
+Address = "{{ navidrome_bind_address }}"
+Port    = {{ navidrome_port }}
+
+LogLevel = "{{ navidrome_log_level }}"
+
+# See the long note in defaults/main.yml. Changing this invalidates every
+# stored password.
+PasswordEncryptionKey = "{{ navidrome_password_encryption_key }}"
+
+EnableInsightsCollector = {{ navidrome_enable_insights | bool | lower }}
+
+# Scan schedule is left at Navidrome's own default deliberately. The option has
+# moved between releases (top-level ScanSchedule vs a [Scanner] section), and
+# pinning the wrong spelling here is silently ignored rather than rejected -
+# you would get the default anyway and believe you had configured something.
+# Check `navidrome config` on the box before adding it.

ansible/roles/xmrblocks/defaults/main.yml

@@ -73,3 +73,9 @@ healthcheck_push_url: ""
 # Bearer token for the Gatus external endpoint. Required whenever a push URL
 # is set: Gatus rejects an unauthenticated push with 401.
 healthcheck_push_token: ""
+
+# WHEN TO CHECK. Daily, 30 minutes after monerod's own check so the two do not
+# contend for the same cold LMDB pages. See roles/monero/defaults for why the
+# name is prefixed and why this is paired with the 26h heartbeat in
+# infra/403_service_probe_registration.yml.
+xmrblocks_healthcheck_on_calendar: "*-*-* 05:30:00"

ansible/roles/xmrblocks/templates/healthcheck.timer.j2

@@ -7,7 +7,14 @@ Description=Onion Monero Blockchain Explorer Health Check Timer
 
 [Timer]
 OnBootSec=2min
-OnUnitActiveSec=5min
+# ONCE A DAY, not every 5 minutes. See the note in roles/monero's timer: the
+# explorer follows the node it reads from, and this check costs a full page
+# render out of the LMDB (a cold one measured 68 seconds), so polling it hard
+# is not free either. The Gatus heartbeat window is 26h, set in
+# infra/403_service_probe_registration.yml, and MUST stay above this interval.
+OnCalendar={{ xmrblocks_healthcheck_on_calendar }}
+# Spread against the other daily pushes leaving this box.
+RandomizedDelaySec=900
 Persistent=true
 
 [Install]

ansible/services/gatus/deploy_gatus_playbook.yml

@@ -22,6 +22,21 @@
         number: "{{ signal_number }}"
         recipients: "{{ signal_recipients }}"
         default-alert:
+          # ── SIGNAL ALERTING IS OFF ──────────────────────────────────────
+          # Flip to true to turn the phone back on. Nothing else needs to
+          # change: this ONE key disables every Signal alert in the estate.
+          #
+          # It works because `enabled` is the only alert field no registration
+          # playbook sets. Gatus merges a provider's default-alert into each
+          # endpoint's alert for every field the endpoint left unset, so this
+          # reaches all ~90 of them without editing a single endpoint - and
+          # without unregistering anything. The dashboard keeps checking and
+          # keeps showing red; it just stops sending.
+          #
+          # Deliberately NOT done by deleting the `signal:` provider block:
+          # that would strand ~90 endpoints referencing a provider that no
+          # longer exists, and lose the number and recipients.
+          enabled: false
           # Overridden per group by the registration playbooks; these are the
           # values that apply if a caller sets nothing.
           failure-threshold: 1

ansible/services/headscale/templates/acl.json.j2

@@ -42,6 +42,7 @@
         "tag:fulcrum-box:50002",
         "tag:mempool-box:8080",
         "tag:memos-box:5230",
+        "tag:music-box:4533",
         "tag:monerod-box:18089",
         "tag:monerod-box:8081",
         "tag:arbret-staging-box:80",
@@ -113,6 +114,9 @@
     "tag:monerod-box": [
       "{{ headscale_namespace }}@"
     ],
+    "tag:music-box": [
+      "{{ headscale_namespace }}@"
+    ],
     "tag:nodito": [
       "{{ headscale_namespace }}@"
     ],

ansible/services/navidrome/deploy_navidrome_playbook.yml

@@ -0,0 +1,163 @@
+---
+# Navidrome + beets on music-box, published at music.contrapeso.xyz.
+#
+# See roles/navidrome/README.md for the exposure and backup decisions. The two
+# that matter here: no basic_auth in front (the Subsonic API the mobile clients
+# speak has nowhere to carry a second credential), and the fail2ban jail below
+# covers the web UI only.
+- name: Deploy Navidrome on music-box
+  hosts: music
+  become: yes
+  vars:
+    # Vault key is navidrome_encryption_key, not the role's own variable name -
+    # assigning a var to itself is a recursive reference, not an override.
+    navidrome_password_encryption_key: "{{ navidrome_encryption_key }}"
+    healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_navidrome/external"
+    healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
+  roles:
+    - navidrome
+
+- name: Publish Navidrome from the edge host
+  hosts: edge
+  become: yes
+  vars:
+    navidrome_access_log: /var/log/caddy/music.log
+  tasks:
+    # The Debian package usually creates this, but `caddy validate` passes with
+    # the directory missing and the RELOAD is what fails - so the vhost would
+    # look deployed and serve nothing. Make it explicit.
+    - name: Ensure the Caddy log directory exists
+      ansible.builtin.file:
+        path: "{{ navidrome_access_log | dirname }}"
+        state: directory
+        owner: caddy
+        group: caddy
+        mode: '0750'
+
+    # The FILE, not just the directory, and this is load-bearing.
+    #
+    # `caddy validate` below runs as root and provisions the logging module,
+    # which CREATES this file root:root 0600. Caddy itself runs as the caddy
+    # user, so the subsequent reload then dies with:
+    #   loading new config: setting up custom log 'log0': ...
+    #   open /var/log/caddy/music.log: permission denied
+    # and - the nasty part - `systemctl reload` hangs rather than failing,
+    # leaving the unit stuck in "reloading" while the OLD config keeps serving.
+    # So the vhost looks deployed, validate passes, nothing reports an error,
+    # and the site is simply absent.
+    #
+    # Creating it up front with the right owner makes validate's open a no-op.
+    - name: Ensure the Caddy access log file exists and is owned by caddy
+      ansible.builtin.file:
+        path: "{{ navidrome_access_log }}"
+        state: touch
+        owner: caddy
+        group: caddy
+        mode: '0640'
+        access_time: preserve
+        modification_time: preserve
+
+    # caddy_site_body rather than caddy_site_upstream + caddy_site_resolvers:
+    # this vhost needs a `log` directive so fail2ban has something to read, and
+    # the role's template has no knob for one. The escape hatch is the
+    # documented way to do this - gatus and headscale already use it - and it
+    # leaves the other vhosts untouched.
+    - name: Expose Navidrome over HTTPS
+      ansible.builtin.include_role:
+        name: caddy_site
+      vars:
+        caddy_site_name: music
+        caddy_site_domain: "{{ subdomains.music }}.{{ root_domain }}"
+        caddy_site_body: |
+          log {
+              output file {{ navidrome_access_log }} {
+                  roll_size 10MiB
+                  roll_keep 5
+              }
+              format json
+          }
+          reverse_proxy {{ hostvars['music_box_local'].ansible_host }}:{{ hostvars['music_box_local'].navidrome_port }} {
+              # Tailscale MagicDNS: music-box has no public address.
+              transport http {
+                  resolvers 100.100.100.100
+              }
+          }
+
+    # The per-site validate cannot catch a conflict between sites.
+    - name: Validate the whole Caddyfile
+      ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
+      changed_when: false
+
+    # ── fail2ban ────────────────────────────────────────────────────────────
+    # The estate's first jail: 02_firewall_and_fail2ban_playbook.yml installs
+    # fail2ban everywhere but leaves it on stock config. Kept inline here
+    # because one jail does not justify a role; if a second service ever wants
+    # one, that is the moment to promote this to roles/fail2ban_jail alongside
+    # caddy_site and gatus_endpoint.
+    #
+    # Scope, stated so it is not mistaken for more than it is: this catches web
+    # UI brute force (401 on /auth/login). It does NOT catch Subsonic API brute
+    # force, because Subsonic answers a bad password with HTTP 200 and the error
+    # in the body. See roles/navidrome/README.md.
+    - name: Install the navidrome fail2ban filter
+      ansible.builtin.copy:
+        dest: /etc/fail2ban/filter.d/navidrome-caddy.conf
+        owner: root
+        group: root
+        mode: '0644'
+        content: |
+          # Managed by Ansible - services/navidrome/deploy_navidrome_playbook.yml
+          #
+          # The log file is per-vhost, so any 401 in it is a failed Navidrome
+          # login and needs no path matching.
+          [Definition]
+          failregex = "remote_ip":"<HOST>".*"status":401
+          ignoreregex =
+      notify: Restart fail2ban
+
+    - name: Install the navidrome fail2ban jail
+      ansible.builtin.copy:
+        dest: /etc/fail2ban/jail.d/navidrome.conf
+        owner: root
+        group: root
+        mode: '0644'
+        content: |
+          # Managed by Ansible - services/navidrome/deploy_navidrome_playbook.yml
+          [navidrome]
+          enabled   = true
+          filter    = navidrome-caddy
+          logpath   = {{ navidrome_access_log }}
+          # ufw, not the Debian default iptables-multiport: UFW is what manages
+          # the firewall on every host here (infra/02_firewall_and_fail2ban).
+          banaction = ufw
+          maxretry  = 5
+          findtime  = 10m
+          bantime   = 1h
+      notify: Restart fail2ban
+
+    - name: Ensure fail2ban is enabled and running
+      systemd:
+        name: fail2ban
+        enabled: yes
+        state: started
+
+    - name: Flush handlers so the jail is loaded before we assert on it
+      meta: flush_handlers
+
+    # A jail that silently failed to load is worse than no jail: it reads as
+    # protection in `ls /etc/fail2ban/jail.d` and bans nothing.
+    - name: Assert the navidrome jail is actually active
+      ansible.builtin.command: fail2ban-client status navidrome
+      register: navidrome_jail_status
+      changed_when: false
+      failed_when: navidrome_jail_status.rc != 0
+
+    - name: Display the navidrome jail status
+      ansible.builtin.debug:
+        var: navidrome_jail_status.stdout_lines
+
+  handlers:
+    - name: Restart fail2ban
+      systemd:
+        name: fail2ban
+        state: restarted

ansible/site.yml

@@ -56,6 +56,7 @@
 - import_playbook: services/monero/deploy_monero_playbook.yml
 - import_playbook: services/xmrblocks/deploy_xmrblocks_playbook.yml
 - import_playbook: services/memos/deploy_memos_playbook.yml
+- import_playbook: services/navidrome/deploy_navidrome_playbook.yml
 - import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml
 - import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml
 - import_playbook: services/headscale/deploy_headscale_playbook.yml

scripts/add-music.sh

@@ -0,0 +1,189 @@
+#!/usr/bin/env bash
+#
+# Add music to music-box: push it, tag it with beets, let Navidrome see it.
+#
+# WHY THIS SCRIPT EXISTS
+# ----------------------
+# The three steps are simple individually and easy to get subtly wrong together:
+#
+#   1. rsync to /srv/incoming, NOT to /srv/music. Navidrome reads /srv/music and
+#      beets is the only thing that should ever write there. Dropping files
+#      straight into the library skips tagging entirely, which is the whole
+#      point of having beets.
+#   2. `beet import` is INTERACTIVE, so the ssh needs a TTY (-t). Without it the
+#      prompt never renders and the import appears to hang.
+#   3. beets must run as counterweight. Under sudo it finds no config (the
+#      config lives in ~counterweight), silently falls back to its defaults and
+#      writes to a SECOND library database in root's home - so you end up with
+#      files in the wrong place and two competing indexes.
+#
+# WHAT IT DOES NOT DO
+#   It does not touch your local files. rsync copies; the `move: yes` in the
+#   beets config only moves things around ON THE BOX, from /srv/incoming into
+#   /srv/music. Your originals on this machine are never modified or deleted.
+#
+# Usage:  scripts/add-music.sh [path]        (prompts for the path if omitted)
+# Needs:  rsync and ssh here, tailnet up (music-box resolves via MagicDNS)
+set -euo pipefail
+
+MUSIC_HOST="${MUSIC_HOST:-music-box}"
+REMOTE_USER="${MUSIC_USER:-counterweight}"
+INCOMING="/srv/incoming"
+# Pre-filled into the prompt, editable in place. $HOME rather than a literal
+# /home/counterweight so the script still behaves if it is ever run as anyone
+# else; for the usual user it resolves to exactly /home/counterweight/music.
+DEFAULT_SRC="${MUSIC_LOCAL_DIR:-$HOME/music}"
+SSH_KEY="${MUSIC_SSH_KEY:-$HOME/.ssh/counterganzua}"
+TARGET="${REMOTE_USER}@${MUSIC_HOST}"
+
+# Colours, but only when we are actually on a terminal.
+if [ -t 1 ]; then
+    B=$'\033[1m'; DIM=$'\033[2m'; R=$'\033[31m'; G=$'\033[32m'; Y=$'\033[33m'; N=$'\033[0m'
+else
+    B=""; DIM=""; R=""; G=""; Y=""; N=""
+fi
+
+say()  { printf '%s==>%s %s\n' "$B" "$N" "$*"; }
+warn() { printf '%s==>%s %s\n' "$Y" "$N" "$*" >&2; }
+die()  { printf '%sERROR:%s %s\n' "$R" "$N" "$*" >&2; exit 1; }
+
+# -n is load-bearing: without it ssh reads OUR stdin and swallows the answers
+# meant for the read prompts below, so "Continue? [y/N]" gets EOF instead of
+# your keypress. Under `set -e` that aborts the script with no message at all.
+ssh_run() { ssh -n -i "$SSH_KEY" -o ConnectTimeout=15 "$TARGET" "$@"; }
+# -t for anything that talks to a human (beets prompts, sudo password). No -n
+# here - these deliberately DO want your terminal.
+ssh_tty() { ssh -t -i "$SSH_KEY" -o ConnectTimeout=15 "$TARGET" "$@"; }
+
+# --- 0. Preflight ------------------------------------------------------------
+command -v rsync >/dev/null || die "rsync is not installed here."
+[ -f "$SSH_KEY" ] || die "SSH key not found: $SSH_KEY  (override with MUSIC_SSH_KEY=)"
+
+say "Checking ${MUSIC_HOST} is reachable..."
+if ! ssh_run -o BatchMode=yes true 2>/dev/null; then
+    die "Cannot reach ${TARGET}.
+     Is the tailnet up?  tailscale status | grep ${MUSIC_HOST}
+     MagicDNS resolving? getent hosts ${MUSIC_HOST}"
+fi
+
+# --- 1. Where is the music? --------------------------------------------------
+SRC="${1:-}"
+if [ -z "$SRC" ]; then
+    printf '\n%sPath to the album or folder to add%s %s(edit or tab-complete)%s\n' "$B" "$N" "$DIM" "$N"
+    # -e gives readline (tab completion, arrow keys); -i pre-fills the buffer
+    # with the default so it is already typed and editable in place.
+    # -i is ignored when stdin is not a terminal, hence the ${SRC:-} fallback:
+    # piping into this script, or just pressing Enter, still yields the default.
+    read -e -i "$DEFAULT_SRC" -r -p "> " SRC || true
+    SRC="${SRC:-$DEFAULT_SRC}"
+fi
+
+# People paste paths with surrounding quotes (drag-and-drop does this) and with
+# a trailing slash. Both change rsync's meaning, so normalise before using it.
+SRC="${SRC%\"}"; SRC="${SRC#\"}"
+SRC="${SRC%\'}"; SRC="${SRC#\'}"
+SRC="${SRC/#\~/$HOME}"
+while [ "${SRC}" != "/" ] && [ "${SRC: -1}" = "/" ]; do SRC="${SRC%/}"; done
+
+[ -n "$SRC" ]  || die "No path given."
+[ "$SRC" != "/" ] && [ "$SRC" != "$HOME" ] || die "Refusing to sync '$SRC' - that is almost certainly not what you meant."
+[ -e "$SRC" ]  || die "Not found: $SRC"
+[ -r "$SRC" ]  || die "Not readable: $SRC"
+
+BASENAME="$(basename -- "$SRC")"
+if [ -d "$SRC" ]; then
+    COUNT=$(find "$SRC" -type f | wc -l | tr -d ' ')
+    SIZE=$(du -sh "$SRC" 2>/dev/null | cut -f1)
+    [ "$COUNT" -gt 0 ] || die "$SRC contains no files."
+else
+    COUNT=1
+    SIZE=$(du -h "$SRC" 2>/dev/null | cut -f1)
+fi
+
+# --- 2. Does it fit? ---------------------------------------------------------
+# /srv/incoming and /srv/music share one filesystem, so the beets move is a
+# rename and costs nothing extra - but the transfer itself still needs room.
+AVAIL=$(ssh_run "df -Pk ${INCOMING} | awk 'NR==2 {print \$4}'" 2>/dev/null || echo 0)
+NEED=$(du -sk "$SRC" 2>/dev/null | cut -f1)
+if [ "${AVAIL:-0}" -gt 0 ] && [ "${NEED:-0}" -gt "${AVAIL}" ]; then
+    die "Not enough space on ${MUSIC_HOST}: need $((NEED/1024)) MB, have $((AVAIL/1024)) MB.
+     The library shares the 50 GB root disk - see roles/navidrome/README.md."
+fi
+
+printf '\n'
+say "About to add:"
+printf '     source   %s\n' "$SRC"
+printf '     files    %s  (%s)\n' "$COUNT" "${SIZE:-?}"
+printf '     lands in %s:%s/%s\n' "$MUSIC_HOST" "$INCOMING" "$BASENAME"
+printf '     free     %s MB on the box\n\n' "$((${AVAIL:-0}/1024))"
+printf '%sYour local files are only read, never moved or deleted.%s\n\n' "$DIM" "$N"
+
+read -r -p "Continue? [y/N] " REPLY
+case "$REPLY" in [yY]*) ;; *) say "Nothing done."; exit 0 ;; esac
+
+# --- 3. Push -----------------------------------------------------------------
+printf '\n'
+say "Pushing to ${MUSIC_HOST}:${INCOMING}/ ..."
+# No trailing slash on "$SRC": that copies the DIRECTORY, not its contents, so
+# the album arrives as its own folder and beets sees one album per directory.
+# --chmod is not cosmetic. `rsync -a` preserves the SOURCE mode, and beets
+# moves files into the library with rename(2), which keeps their group and
+# mode. A track that leaves this machine mode 0600 therefore arrives in
+# /srv/music mode 0600 owned by the curator's group - and Navidrome cannot read
+# it. It does not error; the track is simply absent from the library. The beets
+# `permissions` plugin also normalises this on the box; this is the same fix
+# applied at the door so /srv/incoming is sane too.
+rsync -av --info=progress2 --human-readable --chmod=D755,F644 \
+      -e "ssh -i $SSH_KEY -o ConnectTimeout=15" \
+      "$SRC" "${TARGET}:${INCOMING}/"
+
+# --- 4. Tag ------------------------------------------------------------------
+printf '\n'
+say "Running beets. This is interactive - see the key guide below."
+printf '%s   A apply   M more candidates   E enter search   I paste MB id%s\n' "$DIM" "$N"
+printf '%s   U use tags as-is   S skip   T as tracks   G group albums%s\n\n' "$DIM" "$N"
+
+# As counterweight (NOT sudo) so it finds ~/.config/beets/config.yaml.
+# `|| true` because beets exits non-zero when you skip or abort an album, and
+# that is a normal outcome, not a script failure.
+ssh_tty "beet import ${INCOMING}/$(printf '%q' "$BASENAME")" || true
+
+# --- 5. What is left behind? -------------------------------------------------
+printf '\n'
+LEFT=$(ssh_run "find ${INCOMING} -type f 2>/dev/null | wc -l" | tr -d ' \r')
+if [ "${LEFT:-0}" -gt 0 ]; then
+    warn "${LEFT} file(s) still in ${INCOMING} - skipped, or beets could not match them."
+    warn "Re-run later with:  ssh ${MUSIC_HOST} 'beet import ${INCOMING}'"
+    warn "Details:            ssh ${MUSIC_HOST} 'tail /var/lib/beets/import.log'"
+else
+    printf '%s==>%s %sEverything imported; %s is empty.%s\n' "$B" "$N" "$G" "$INCOMING" "$N"
+fi
+
+# --- 6. Optional post-processing ---------------------------------------------
+# replaygain and lyrics are auto:no in the beets config on purpose - on a single
+# core they turn a 2 minute import into a 20 minute one. Offered here so they do
+# not get forgotten entirely, but off by default.
+printf '\n'
+read -r -p "Run replaygain now? (slow on 1 core) [y/N] " REPLY
+case "$REPLY" in [yY]*) say "Running beet replaygain..."; ssh_tty "beet replaygain" || warn "replaygain reported a problem." ;; esac
+
+# --- 7. Let Navidrome see it -------------------------------------------------
+printf '\n'
+say "Restarting Navidrome so it picks up the new files..."
+ssh_tty "sudo systemctl restart navidrome"
+
+say "Waiting for it to come back..."
+for i in $(seq 1 20); do
+    if ssh_run "curl -sf -m 5 -o /dev/null http://127.0.0.1:4533/app/" 2>/dev/null; then
+        printf '%s==>%s %sNavidrome is up.%s\n' "$B" "$N" "$G" "$N"
+        break
+    fi
+    [ "$i" -eq 20 ] && warn "Navidrome did not answer within ~60s. Check: ssh ${MUSIC_HOST} 'systemctl status navidrome'"
+    sleep 3
+done
+
+# --- 8. Summary --------------------------------------------------------------
+printf '\n'
+say "Library now holds:"
+ssh_run "beet stats" 2>/dev/null | sed 's/^/     /' || warn "could not read beet stats"
+printf '\n%s==>%s %sDone.%s  https://music.contrapeso.xyz\n' "$B" "$N" "$G" "$N"

tofu/nodito/vms.auto.tfvars

@@ -89,6 +89,19 @@ vms = {
     memory_mb    = 2048
     disk_size_gb = 20
     ipconfig0    = "ip=dhcp"
+  },
+  # disk_size_gb here is the music library's ceiling, not just the OS: the
+  # library lives on the root disk at /srv/music. ~45 GB is usable after
+  # Debian, so roughly 300-450 albums in FLAC. Growing it means a Proxmox
+  # resize + growpart/resize2fs (03_vm_disk_enlargement.md), because
+  # ignore_changes = all means tofu will not do it.
+  "music-box" = {
+    vmid         = 8200
+    name         = "music-box"
+    cores        = 1
+    memory_mb    = 1024
+    disk_size_gb = 50
+    ipconfig0    = "ip=dhcp"
   }
 }