arbret: our side of running it on vipy

prd-arbret was folded into vipy. arbret itself (app, worker, Postgres, timers,
its backup dump) is deployed by its owner's own repo; this repo owns the
shared host's side of it. arbret_handoff.md documents the split, the Gatus
push API and the backup contract.

- services/arbret: arbret.com -> localhost:3100, www redirects to the apex,
  port 4117 allowed on tailscale0 only, /opt/backups/arbret pre-created to
  the contract so the pull job is green before the first dump.
- headscale ACL: framie may reach prd-vipy:4117.
- vipy: arbret-server, arbret-worker and postgresql in the unit checks.
- 402: DNS and HTTP checks for arbret.com, www, and /api/health with
  [BODY].status == ok. Entries may now name an fqdn outside root_domain and
  add conditions.
- 403: probes arbret-analytics (hourly, 130m window) and arbret-beancount
  (daily, 26h), pushing with the dedicated arbret token.
- backups: pull from backup-pull@prd-vipy:/opt/backups/arbret/, and a
  backup-dump_arbret endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

02_vps_core_services_setup.md

@@ -285,22 +285,6 @@ LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd n
 * Start it up again.
 
 
-## ntfy-emergency-app
-
-ntfy-emergency-app is a simple web application that allows trusted people to send emergency messages via ntfy notifications. Perfect for situations where you need to be alerted immediately but don't want to enable notifications on your regular messaging apps.
-
-### Deploy
-
-* Decide what subdomain you want to serve the emergency app on and update `ansible/services_config.yml` under `ntfy_emergency_app`.
-    * Note that you will have to add a DNS entry to point to the VPS public IP.
-* Configure the ntfy settings in `ntfy_emergency_app_vars.yml`:
-    * `ntfy_emergency_app_topic`: The ntfy topic to send messages to (default: "emergency")
-    * `ntfy_emergency_app_ui_message`: Custom message displayed in the web interface
-* Ensure `infra_secrets.yml` contains `ntfy_username` and `ntfy_password` with the credentials the app should use.
-* Make sure docker is available on the host.
-* Run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml`.
-
-
 ## Headscale
 
 Headscale is a self-hosted Tailscale control server that allows you to create your own Tailscale network.

ansible/group_vars/all/main.yml

@@ -36,7 +36,6 @@ subdomains:
   lnbits: wallet
 
   # Secondary services (vipy)
-  ntfy_emergency_app: avisame
   personal_blog: pablohere
 
   # Memos (memos-box)
@@ -62,6 +61,11 @@ subdomains:
   # has nowhere to carry a second credential. Same call made for monero above.
   music: music
 
+  # Jar Lid Archive staging (jar-lid-archive-staging-box, proxied via vipy).
+  # Public with no basic auth: the one-off photo links are for people without
+  # an account, and the app has its own login for everything else.
+  jar_lid_archive_staging: jars-staging
+
 # Read by plays across several groups, so global rather than group_vars/<group>.
 headscale_namespace: counter-net
 

ansible/group_vars/all/vault.yml

@@ -1,214 +1,229 @@
 $ANSIBLE_VAULT;1.1;AES256
-66356262646437623239383039643762363139653565356263623438303833663431396566353163
-3631323563343638346464386661346235656130383238370a353236623366313561646334363639
-35346232376138666430363437346638326264646562386436323962633630663136626462343762
-3862333139646636650a656337306533313436333239653561333231363030633339303835396637
-61633763313464353933306333383061346563623637383166646332303237653065346233623134
-39386130613866386235376266643062653833643132633235393931636366376164306331303762
-33393562303738393364323465653163633730633636343764646337383733643362396132303063
-32336461396561663063633137613234356535383863346334666465643437613962376439356433
-66353532333933343462636163353637613637356530333331636363616265376263636538656566
-62643137623933363237343865613131383766633561636664616434393538646332346132336630
-62333361626339626638333666393735373834396431343965386532313834353837336664306530
-36303138373734343735653637663466353435306166316661353738353035366535333265626330
-36396538353566316634376662383564323861303063386163663061323435656261316439336539
-65323063343961646435373134303039373139333265633838313636373463316231633832306261
-36623934343731343938306332356261646264613063393834336332666230626534653464363335
-66356331666434623735613333303466656263313964363030316231303330303735373135653763
-30323635303435656639653037366430316330333033306339366361353765656337316561366263
-38613265363565373832636239396362616435303536646465326132626239353361613137626632
-30353764323430656631366134373263343861376666346166363131333137376466393438346461
-31613435363637306432343437653636356665373838626361306332316664343266356666353238
-35303464326633326438353730393462373437313864326638323262323034346464643038386462
-34613435326432663566386331333430663336656239353131633261626637613035636662346136
-33626131626435316566393831333534393663326465633531373932626631613063666165613766
-63636463656435386333323037656533306331646561363061613537663738326162656332393161
-62313338376630643031666630663337646439623862306663303335303232306161353336643334
-30386537336265383334336431653637303836376261303038653739396263666161653365356133
-62636638393465363239616264316331316139363063313431323165313237646437643361653539
-63393932623934313062376137626266363330623766646231353561646332353565643638356633
-39656565363962653038393430363666643361343131323231663566343035643631646433306464
-64343735393937333036623139303531343338373162396635613863316466393235343865333065
-63353662313737353737353736316261346466366666666536653064356130326334326137383933
-64306331663430313838376266613763343537613965303937393364396238663266363539313531
-33616133646365396336613234636630396239316633333832653138393964616261643531663836
-36666265386134383430626637643766636439393636336436366265396239303335306563653962
-62323763626265373037343766386433646637633838643933633738333037633336646663316335
-36373534663439306137353236343565636538636333313239373936653263366637613335663233
-32306536643866656461343666396565373661353561393665643162613539633634666138626266
-35613663376465626363373864626266393335396461316338653262316131643633666361333537
-64366533633535313633306535393732383436333762616539646463366665623866323463336263
-38313365326338386664363538633232343636303135356230393065613238386366323435633931
-37353537313330396463383233616532653139376263346264623366383339306139386231636466
-35383761373733386133386436316431613265306264656630353561333736613734653164333031
-32643237366534643062653065333632343563633034663266393961353763663534353466383561
-38373332323135353932343231353433366637333437636238623235663836393735346230623438
-37616639643231396564373033636162616234366262333461343139363539316332333431636565
-64396639383462386239613035303565376163633637653137373631383864353532373465626630
-63636531653063306131623735393963623836313566643565386336636565633130383264316533
-33343030316365373164663661306161363062646361646332393935373533626331623163323038
-61316666366263313531373530366639333364336563623162323938356662626565383363303533
-61323730313161633035646435346234366661386339346663363438663862636361343033306134
-66396365653634373235623837666537323230616334663263633135316433313734663863333561
-34326134613335346661316632353764343235326630316535306666326662613436383764646366
-33616135333038333834656439383733656436663433303536313961333662366564376338326661
-35393839643663333130383965663538393931363662336165636435643132353461343266363265
-66346635313337346661646461386336386461346133323435643939616234666665616261313731
-38383265643830646364373164656133396132666236313265346337643939613831303366393530
-63333330346131656134366237323661316534323330306330616564393130353136306461666361
-62376637646230623264363437313161396130616231346133346665303834623833663062663436
-30643736376638303065356161366561353565343139343731313263393933313239363363636638
-32663932626332626366333635616335316235363561373539623265663733373030373439373831
-65333763653635613964396437333833626535643666643433393031336461366562333934666234
-62383632303436633965643430363763353361613762306361613066356136383131383232313961
-30626531353933336563656139316338616565303261363933306164393966343037353431653233
-32636631373966323838333666313964323933306638653162646166623237343662646134383337
-37383039383531656337626334386638613133343133633637343738343731326663356666333161
-61323765636135306563323064346465633537613361383761393361353363396237333438306166
-35386338356135633365396661356535623232613765653931303063353538643130323439366438
-39363936316263333562646133646464346362343761336662353764613261396364626164313664
-31383839303361323461663662393736616363646232666633323134653138306161323735386638
-30333365663064366535656134323338663566616163333634353939363533356364393563363333
-65623838323034336530326363383237373563326334343066323738663833353930663930373965
-30363766633137396561376636636566363862353062663038653534623430643766633631356562
-63623762343433626363353239376539633136333038623338303961373938396632313966666532
-33633865626462636137343436386661353561623035656435623566613530623163316465383664
-37346261616239653138623262623930316432633531313037353938396164646534316564613238
-36643137303965316238346134323639386261633338363130323134336632636564636638653861
-38346566643966386465366535393166626463313039613236393831393732613537336530393936
-31613234333836333732323165653931336565316564346239336663363736323763623962383261
-65626332626565636162336264303964323139303831656262373161303861393366353237393136
-62303361636131646335633733373966626135396630623133623733373565306234646165656138
-35383065613365363438396332306236646432663731326666613838643434393866623163316631
-33633931373135336438643735646266383462386431666330363066386538386335383231653334
-64323366363032386662333662613562333864356238626161366566356564343339343838373437
-30363133363136663036656461343534666636343932313336336637376665393564313234373430
-39356362616539366237646363646364326664653232313235653831663539353530393261346532
-38653134623732313466386635313730303639613561323665356633653836393237333737323864
-36323736326239636565346536646566656138666630396366353930383237333338626532663131
-64366339303230663530626632623066613030663533366539313866646466373439663831363936
-36366636653032343064383661383664636463343165326532653861663537633238323937363839
-65333030653566306465643035633162393534323731346330333861353236393732346130313034
-64303736323030643533633539623638643232306435663238393566376564313162633334613462
-33363931383664633433363032643536356135626662653733303339656339373735643732626535
-35316131356135393933393462336163373062653133356133393830393638313838396361336635
-62323463333464353939346233333462646161633036643332313337353065376561336630383163
-38306666666634393766366436653436623138363830343636323833353365333762343533633835
-66363730383932303661313831663566613830393530616164323963636339393339623065316534
-38373662636330336632653335316463623935653961363566396132616433633831636334306236
-34393761663962623563333534383739613965353033646235653532303330343961656639333337
-31643462663538313930383038343964326463363935396365343466643236653738626639616434
-62343337356239316162383564666131663535313633373062353030336439396533376235386131
-31653865643931323364356566316435333336316166643063613164643266313834323235353863
-34623936613366303161353165386662376465656366656561613466383830326162336238656236
-30646632396231653263373631346330376461303231373733646231373462623131303633616166
-37366435353830303234393962663334656335356661333437363637663664323430623135346262
-33303233343362336261633663336635366233313466636233333836663464633637383964313133
-31363461336237373337353438396132616134303432336533653737333538383534363735316334
-62303666363532383131383833383536333733306163366339653038313930646531653565386335
-38313561303062336232333634343866393763636131333665396430306432373734613834316636
-61306138626530616230343462613036323166656334656637616139663030623932386132393762
-36363364343737613165376365323361363233616532643866303632653063343431613338373633
-63303263626434326539316439323239616632653632346538336535373336623739323931623462
-35656365336630323035616161656237353135373366376466326264323863336166633438663834
-63646562386339363235396163646334333238323932383230643964366466323064313138323464
-62636239663635336332633334656631383261306437333963363464303430666462633333666638
-37633830663439396133323861303735346561376434386131363864316332623137376139376231
-30303765646132306261363335366565306564373261663336613336336632616231363366316135
-65303966616439643863326662656439623938373833323738646535636230663136653266653533
-63326235376633333366373436333534336436366434363461393763633437616262326133623563
-37613161303161356632316335616639373564306161623339353066383163303431313333306635
-36326435376661343237303633316462613532313232643461313065376134393338613130336161
-38326463303562666662613361656161306139653431386630303561626666613037393238313463
-62623533396532326332326636346134616236653638363837643131343166333564323538653638
-63383261383265353331663864643933373562396565346163396434383765343962336363353261
-38353034643436626632623235396261326235323937646135326532313561396361363062333366
-32343666343163636233646466613633383034653438646262666133643830333834383233343136
-33646437396137613466643031323538626332613066333065643237353263636435393964356464
-34383265303364663233326432313230336436373866303435336666343832383737303030306534
-36383036363937393965386431343561333961663565333565333261323033616630343130613934
-32383166333063623235333464343864326531333165346261643131393264346534393035393431
-64636139383232383332623866323363623666326361386134353030376366323039393033653061
-63383338303734323637386366646365663032636463336261643533663638343661613462373736
-32663566666133656165396233643334343335333731333938376236633330366262366139326133
-32653163343963333739303061643936363861313938646164316564393261663765393462613837
-31646636326330663464363034376230393739343961653130303166366233396262656634313061
-65646664373161356134616165383265346633376136376132643234623334353563323539333065
-61623265343730326164373063383937323766363466343466316364373935626138343938323261
-38323739663131346138623663393832346236356631633133313436363364663536316561623631
-32393562646666343537663039363165646337653564363065353433643031353865386361613534
-37633639303636313261613563353136346465326163623361323230306364396362373966336262
-31353565623239343564333533656236656666316638646666376138373164663138633835366162
-37633036346562663261343731366161613361656534343839303062373232636339643961666432
-61396665656638333435386338383638323664313139656331336131326563373235646539633835
-30393235316564656131303932386336366661373737393264613364663030363862656462646463
-33303766303166336132626265653431313263363131653835336634356632666230666235383262
-31653837356536666266383562633437376161653664353533656331303563626539626161653562
-64333337626430616330346233613566353330383361353234376236656336653735663838343333
-37643162376630373531396138643861326562323133303363363837306138373434306364356364
-64623037643737663138626262326630323638376336663062393930663566316632323862653135
-66363664336431373966663133353030393863313538366632643932363438613035326563323163
-63343565373862316230643166336438393464323032666334653631393761303337303861666134
-36663637346136623564613233343432383930616339626161623933323565623861653939326165
-37383436353533613736323237666433316130613464396333343965393031333138396461376632
-61373166643766376234646436653566313637326434363261333831336136306137333061323035
-66303335363035666366386364303861383231366339643666313636623733616562333539643465
-32353566326262326637393434656133313963646435333135613630313865373430366265666632
-37316539363439663131323835313761313765653564663337346462653332613430373037333064
-64323731623566326664613634626265383661346462323463303638346464643366663333336339
-65626265663230346338633764633436653339373031623438633638326137613635633239343834
-35373132376133363861373062386235353836633030323236343666323134326135336231613566
-34326263643163633935643065306165643564663831633561353061653331326566633664376432
-33353835366265346266663836326465356166626432356663366566313962663862363930336334
-65323337386231376361343933353865633330613466323337306137633762653931313331313339
-36663965353832386662666433383934303737656336383432633632646135383835336631373861
-63646164373137316662373330333465343134633466396566333334663036663039373934333832
-35343661303565646464373834646234623837363562386463383237373533316638666663613633
-30313437626231343062636437363836386635333862373362643532646335353863323936663034
-30316461373330623130353563643035303561303666393464636533626562633165393236663936
-31313061303465633332646363643866306439383636356366623565663437313365376331656666
-34333930333235306636646664326561613165633233353937666238646536323131313232626366
-36623237613635343134313366346661656433643136353433393532303165326564646331333035
-35353365633039353161663337306463303535643062366232336239656162653532306539333035
-65373735373832326337333962643330363338656134303637313337393466366263363264343631
-30656433643562393537326361323733643639396537303065623936336230346664616534323262
-36313864396364353836373538333130303731383437376137383938383633373063323936613464
-37613166323432363232313834326565663265336138396335313230303935396266646438323833
-65306666636338373239346332613464373235613137643133626261313031326462356138356632
-66363465346165306533613636383938636531656631306139666237396632646634333937623032
-33613439656637383537356364336265653931613835333734383836343930313838626564333065
-35363735313162616238643438316535356663396339316331633232393232316131396432653639
-36303335643534376536383331323562353139613130383330636334636261326462656565643138
-63326365363163383932653562363833643761653839306131343737353633313737353732633463
-62326563336339383661376665353230613366646132653364396338663337326136333263363838
-35626537666238383435613431666365303133373166386136353061373765313632343665306439
-34356139363636623639626633623030363933303539626665383733363638303832333561653130
-64393437386362333731643131663263626434666661326533386362306362653732376366626139
-65626662323232636134653334366639386164653535323435613863353332366630303636663339
-34333633373737333732363664353437613762383064316134393139393539306564396237343332
-64636339643030353365356136366264353935643830633237316532316265316635393664326533
-64343030396334326135336465336637623832363065353163316465303537643861306632336535
-31366338323263313965643565656430366566393130326333383862313239356437623232613865
-63383863316639346632323834633739356464313865376463323633393365653033376431373138
-61363035653233373036666531303331383337353839363563386333313730363664666463333735
-61383638363164336265383664376463616536383533666265383961353961303833353265366633
-32646330333336613631633830633837346332653437356536363532396438326364623465383133
-36656231343734643830383335623930343132386336666366623161666637623862363033373236
-32666332316266623230666231663162643262653231313965323137383532633663323239396139
-37656533373065623161623766333335353965373731346639613566343236363239313537636239
-34626533346138613764346161383337623265353563343066303236333534316137356636396565
-32616136306665346566616565346162613736333836623363346232373932366365663531653435
-37623539366233366539396361386163643161653731353232313034613135373039653339346637
-31623138326133366333656661393837653939343064656638616630346462363065623139323131
-65363036383433663566626336353432613535363239373264356237323032323634383662613763
-39646161363639326330346332353965366330346632303939623136303238323431346361656462
-61623838626139636163353430636461666239343337373132643334616234616339313761353331
-36643165373834373437363165303861306135383236303030626463373235393234313765343563
-37353932366430666539623230373161303935633565323366363232336264643961383835333066
-32623366663965316662383036306462616465353139336333643936363838313538363832363330
-33313861313636373831656431383434363664663734626363646365666135376331366335653064
-30386235313037363365363232316338613634643235383037393635333166346632356435373039
-66663862373939346434333130383065386135306661353066343330666563386136376562316134
-63346435643132316434363666343964303335663131383831373430653632663638396333393932
-36383664376666393761643730306336333538376637633135616166346432636464336434323031
-3966
+62636131396363616337633131353962356639613766396163383633353937303334306661656662
+3734393736396462323634396236386339623336663033310a366563623930616331616333343337
+35613165633465653630626139393930333263323238353530663065613734646334663833323638
+3364613961656433360a633136376664356262633338356163363631656665623562653034376338
+33376661633533653766636135616261633133313338383238313335363461373064646261646262
+36383030393235653566666537373232666538616435356332373530323536386535666532656461
+32363638613131616137383133613966353232323537373938306232643738353163306336613238
+36396539623730333232653338393636333464353536623337323935613461396136313131326133
+35646561323165323431633039366435623965396363353535343165656631626166393234643639
+37396139376531653839646233313837373136613337326439373062393030393130623638633334
+30343266323938333333633064376632663737343332306533653232653837376637656166393935
+32353231333532326436383532646336356162333738333037336337363133396465363937343330
+38333335316663333232663966376234323261353036363262353362643061396533646139633539
+35343065653763393032346333373734376133613039396161343438666566613530653463623239
+63643762663933653333363665383634373762303462313064323336353862363530343831643336
+64653030386438326633356138316338353838316562643138353065623262373639326638646365
+38653637633731326364653834393132336637663135366230316437316365363366343530316236
+31616466383462626266316330303532346536653435643234646534373064313564343661383139
+33363864386266313563643264306537303066366463633062346237653963383131633436303234
+32613462616261623632343566653836623539666539643865656366313264616635616638386237
+65396135303739616635356332353336303861373965323834303261396162393235663963383963
+38376136616537396162333566326465323761353030393961623262336165373931633333353330
+30313137326661363137366265376132333063326130356334616537613330373664336462666137
+39366336356332613836343039623330343231613530373463316435323034653732333362336362
+65313763393863303935353430333430333632633036363932396238303464353465363032613536
+66663766356262313135653964636437396665323461386337303731326166386331363134373133
+65636436323336393034666638646632643336363031306163643266323333633031653734376662
+61313734666436343062373037336262633333356462303562346536663862393431373334323363
+32616663383135613933303430323030656130333264343231393164336536326464623964313131
+38366139393732376665373537376563656332626530636363613433666237653835306433363031
+62323734393933363064636464616132383636353965366365386261303530646233336266326366
+63663332656233393232376235373836306432373434316532356264613733643530373230656135
+32626463613663653336303064306563323133383834313936316366623931396537396165333563
+31633264663030643164353138653565363533373336333833386330313335636230353030613061
+39333462663965333565363730313731623538306436303337653361663035386130333665623463
+63643536653831393361336665616361653764356163363832656463363038343330636239356136
+33316238343538306538306564326337396633666166333863646462326164376365303836323530
+33636230376635323166343631373530623437366461393638383532656564303233383865626332
+65343063666462656236346634396464386437616332303437333932663966333162623463353733
+66646563313363633361393862616563353232656136376138306239383561386636333231373436
+30333336363831666530363132613862306334656133666231633063313263373336373562353238
+35323166366163343430323439646164646161623264663966643334653237366335666464373131
+37303663643638316334336665303562363266373130656230356336303531323161393365383430
+33366165613236363138636237363766366366363539643361323664613461383965616333346661
+30653436373030333232393366326435653061383963316632613163346165633335643731633961
+34396261663939636237383439613461303735393233653164363837333235303132626436626132
+65383661353261383437366235316533343664383033333966333531353131666530303439353464
+39396337373331663634333839656362653761306130363136353861653033303430323938663339
+64376636393935313633336464333930326432383164376139323533356438393761376462323764
+35313339363032373463613335666438306139396634386466323039663735643563333664636364
+63623636333837366165373637613431613962343966613934663065383333353761336331343466
+31393466393661336539383161613963313661313366316437393835303564393335336630366434
+34383434613637356232383863366465613133303964393033646239303932663563333535333931
+32326630336233663131613063363533333133336463366466313362333331633430353433336131
+39346639306630326336306537663262386264383938643362633762306364643435303533633635
+33333364333466393038353434363334633965336533313135396436336432393333653832363531
+38356633633632346263663935326264396632626263326663303861306363626134313761396233
+31396435653463303530633739323930396438366166366137386566346338366566393961643865
+33613365663863326331616163326666346633663136663330623331373439343330643261623665
+63656439653935653637353134623865633666343136396361363835363861366162323234363737
+62333135613539643261363462373534623631616438663035396162646161373665383364656438
+61653337316537336337313231326639386665636263383566303164353535643238386563366661
+63633037386536656332623331316361613064343833623766366665643232636265363731346538
+30353037393365353333393261666532303035393665356165633263663066623464313935343536
+61636462613738623861386534353332613734643534383430613564616235313432303566633963
+61616662633638336261326365373735343264393432323965316330373430656635323937346239
+61363433303235613630666331626361633938376564353831623664323334626233383133656235
+66643732613432376561636436306236663734316136643735376564383362356262666131373038
+66393931623637333237646332356531326533613932393232306461623235376664383334366562
+30356365613331646530616431306330363063623234396265633936343832663036333364646337
+32376138316533643839323766613838323265653632643430356234643665303666386664323031
+66343764643038666237343536393737623862663363666464396535336335616136373930336433
+37303862343834306266366335623036653738393234626434343337646338643631313662666634
+64383034333936643164323736303639383237623235356639383165646433303935343062396633
+32633436396530396330336238313466316334386531383962336165336461633836643963333164
+61663434326463623737333530353965366361616639396238386565393533363339333235643761
+64663230343964313435663335333039616165623466376638616534356638396538346234663436
+38636233643265353561356666306165616462373639306135393866386433366536373938353764
+64613637616431623963393534316631616439383161373135393161313332653134643237353037
+64313966636566343766326132376336623065363937303633363634356438396161303437656636
+34353562316663666430396630343665646165663061633235383232323363636637383465343538
+37303733653166316564666163333963656265623264656666613539666166636466333866316634
+32383466623962356338353334613166363932633330633836353037336139383732613766653030
+32373064363766303436643331363732326665646334316663323562376561353063663837346537
+39386531653762393263363261353136313961626663643865616266613834333664376664643761
+31393661393634383030643462643062333635366135313564666336613965623730313138646365
+62396565623636656335343737303732386161643537663236393831336336363764643762613164
+66396262363265386164346665383961333332346139363337396639343731346131373932653332
+62386564343639343930306164376432653961386165353333646134306231383334316366613337
+62326339366130653333643333616636653937323130396563636666373832383066343666376465
+37373932326538666664313534633661373035333765323733643064373865356338376130626137
+32373763373734383539313839666665653536356335373933626361646139303235656132343033
+30346435396637313234346436303964393863303735323864613038643136343237376138346334
+31333637623339343263646533383434393862626134303463663832613836373565373164326435
+62663461366639353537376664313463666437336264323165333734386362313430653839353437
+66373130613338333034333663366564616562363632336236363237653536663963326230633136
+30383635356362616263653637356234613537363364313437636662623038643537666431653533
+35383165366537663839633638663938383836616366313831643630306435616431643464393562
+30613634303266383535373965643534623338623466633365643839396564613236663631343439
+38346164653764383333393432343565326537313639336630366163653937343566303938653665
+32316262653630623638373964623430633466333835366439666530356535636630643063393562
+37643966346437396463383133306466336365303137313932376536666462313262343131623161
+64376261633964623761623666636134313038396564633436383839613730396139636530366333
+36346338313330353437343464383232666338613337336464653663613662383230643939623136
+63383163663062643436326166313432306166646264623933363935663862373737383434636234
+36336464353934383034373166373733343133353165623661643333663332303266643538666530
+35653063366463333961303761633763373266636639356239653636653061303865636332636632
+35633333623236313566376131333830353734303665383433366239353562666664643662323065
+35363164393138636135663462333131313632626535653239323463653634623339393061663861
+61626531383964373730396536663434306531383437333335316166663864653561643939643837
+30396137393665316239633330343161643562353735316330346565343366616536613330306664
+32343537363466386664353638623034616631346366633862353538386566616137653162303466
+38326531666130373864303235636261616565386264663335663862616265666366333739326631
+38323766663864353331396334613735613436363263653938646661623735393038393661613266
+37313932313434313735653330666639623730666462396433356364393931313632383634353334
+66346539663836636139393839653237623932333537653666333832626333396235383731373237
+33616634336662396538323436613166616339636235663933313437353737376539663438343164
+34383138363833616364393537333331396566643562346464373134313433333931343939336261
+32653231656265326662653639376365643234383265363466393864363562633066376664373261
+62343233633034633261613839333461336639643136656166326534383366343239323262303630
+36336265313233386361366139623663393638613538313265366462356463633862336364623962
+31336363653964303466626239356435666562386535663832373333353062343034646432383832
+35383831383331393962376665613634303634333864613066323530346134303934386564313666
+36656162343965306461306132626336353435643334303531633736613633646536353433626437
+63373033323434616539363035626632653663613838653232303039356633636264643432323236
+35313461616231613563393436633933363935306236316466313166636564323139393137323534
+34613432306237356663356434656331626161663138663665613663313263366162623561616234
+63663865623765643236343035353739633266623366393139333261353539313462306132326463
+61636665353861663838396236643563353134663831643061313234653830643864386131366562
+65643866633632353932313733363163373237333635623736666139393766393634626430333039
+63353536383138313165366362633230383137626364326333353037353736633664663738343766
+35653536626531393832346634666131333730643465656136653433373134306461383231613963
+63393366663137653635383830613239333337626432666436666363343637346331303731626137
+32326366643439333966616165636632396366383364313832356636643032306131396534653339
+39383732633132646464376233383139653865393061623763313865343964613031643965653939
+36633963633033313530666661363331353836666137386536396533306165646131363235303336
+32663537316164653935663034313833633135333837636331303363653864313231353734633461
+31383035376639393335346531626266323730363934393062366263313136343665303036393933
+62643965653932633334366363633763353862333335643265386533383062346266636639653064
+31613464393866323363393731363766363063303234323861363166373430383266663234336332
+61386561363562393736343939356434386464626134353634303931396662313665336531373235
+36366463623562393139383033303630663062323564626135613837623335633763613435353964
+30323161666264646630623532326431616135396133323838653736356338616339656638386464
+31363331303164373031326161353666613337653536656330653865653062316636656237376562
+34386431346330636464616136336531333331663865656266333964316261613265306434653038
+35333935626461353033323733326634373036663133326336373665383233343636396466313566
+37396462363764326361363561323633633537353165396362366166643362636639646536633065
+34643633383264663137363264663964626335643938333933306232363466346430666230353561
+37643836623262343664323664646161353230393435393033306536393531323166363739643432
+32396236363839313131306266383866323861343537373738326265326637636531626431373335
+65386564333961346633663633313737396565383431373265656131666638626462653335393330
+33356362613535313063316330306330656263353463313538643232663136636136333539363663
+30363066653633306537626664323339326565633134373831636633623930343635396362393561
+35303237616335653263643132386237386639373939363536356238656163343133613065376531
+66386361643465633938353438333730323933366465643533323738653762353437353136373134
+36636533313665633362323361373839633839653338633336373561386663343361313733393031
+34626566386635346439386564643261373363323466633839316564336137333636353364613062
+37343038373933316665376539396263393765613034306139393962346563353065316638613738
+66393739663736383265383735393331663739306462393939626661656534366665333132623831
+34623962393535353665343537643938353033343634646664323535393762366639376132626362
+34373230663937336161333934333935343564336237313837316363623665313564333134393133
+64653562313036303137316436633065643062303335643266366361316163306433376565363034
+63633830633937343530653537663465336563373435376366623537383165663033623539323333
+66313238363266653065623532616331623635643038353032303266373565646136646538623530
+31613038646331313738373634353737613435303964316161323938316261303237653362383233
+64616133623432353139323835366238633562336161626564633530623035336232346465386332
+64343036366430646161633461306661663564343132346564346633303934663539643239323536
+61386238663037373534336535396230613565303566626631323138643365656132333635613532
+34613633313633366638376662366464333664636538373133373939613565356637383863613035
+30353261643161633734386331623765326664616636333965363161343661613937356636363864
+38633132326237623964643734613165613133633832396633393432303535623237663864373135
+34643964636565353635663633623933643164303734353766643665313933343565373530333833
+36636165323030336562313764633039313165323630313665353366313135653636313338663233
+66663431346234333364666661643165653963616230376130663062616366383238666561326364
+36636530333633653763363435373534616435326533383030386164646130646164323262663330
+32323163623534613036316134366463616639633934396363653539343830643239396633333064
+31323264663866353530343033323066393332323161363031626136336531666566343332663230
+62653635313763313461376236663432306130383264313763356531646637363432396365613766
+37323935623031656438643330616563343632376230666236356534346466323533376631346564
+63333536313132393164356337643465373737613939666666363361303636333436306632626166
+63396637313235396236646237383162366531303133343962333639326139613231663737313532
+37373732383734646330326662636330646135666462343133663336666462643264313565313933
+31373035626564396665323235616138326539336133636232633034383262396466383138626264
+61663166356162616465353565646166636238376464383033303636623334666236396637633635
+30643861623632663632616130316630333234666439393030303961666239613530333937656563
+30356363313733393663376433346138383966616465343934663734666437653863663133643437
+64613236633736333237383834323338313431326634333730656536323138626438306231333035
+35373936646264363337613134396337613039366363323638656437656564623831666461373064
+66633232623666383738613834663938626438653430376534306164646363306663643362623137
+61376433636336346266393532363131613938356262303564393636613833643738646230323938
+32353763326631656561666533366236656134343638393435666437646231633463316232666563
+63653635393334336266653265326239626531323035653930323438393235633962386438306133
+34333537326530383363343139363461646339383064396637663465393961306139643761333538
+39346362663830373535386665386533643435666438656638373861376537383538366331323461
+66363138326432363432343335636434656139386161313162643838343566356264313734613630
+64366162393932313131306533633666393564613835363664306639343161643339316537346338
+64303764366136303737666464393463343261383538333563313164343666383132333761306463
+62656138633939383239343533356136383735663834323763623437386434313333306230326237
+38373666376332646635366233646237373431336531366432323130353034666265303236363937
+36356536653465326139306635633032326535373734333664656339386266363365303139623463
+38396664613639613737343062326538653835373861393939656636366337366134393236346463
+33306462633639626463326230656338636664613730373731323531636337333062646233306139
+61303661313764623565383265353061616638663636336635356364383933313363663330306638
+39613637363935626363633635333533323735623963346334303661353030313066633636396361
+65303738383131363237653866636637346464336334646362326538303232333661623038366332
+36323736333538373931616638303030613765326165663066333131373734373766343066626131
+32626462653538376431373635363835643036376230623061633963393065323262353931613165
+39633136656666653231353036353034333634646266386533316232303462663161613239383931
+64623034333137376134373235626165356466323931333131376238306536633861366430616362
+63636233653339376665346635656239306161663538656166363563326564643661316563346666
+38666161366430663234396237316534323263363436613435643436376535643430323432333639
+32373436353936623966626263653037366263646232336665626533343366366530393535663733
+32616639386433303032376565626535393466356136316662626364376666373934613166303233
+39336136666437323766336430653433393134656230663961666239306639323039396432653936
+65363736343637636538646130616362373965376437333762383138393134353332663734326565
+31303136613739643965623437643663653339666638363861343164663538313234313330663738
+37646362636130663431633762323634613964666166323962346234643765336639613934393938
+64323239343832326564333731663961626661643936643233336662303335663861653331326161
+64396564316465353163633066383562643636363562376136363733646439356530643432336139
+63366265353435323565333161386563636134306261653366623036396266356137333364343930
+30313465313834623530663436623333663262646331653832656266353834323463666332643130
+32656539393838393766643962396133333038343166336166336539333434363433656265313339
+31613238643339656130396461306134646666386430313636376633333465356230333433303666
+37313534343037336432373639303534633235623264363064653635333837656132643665396132
+66336463313139336163623637383164643237636134653837663062303238633237306562616161
+62613536653036333431346131363864393730333432383632333034646361373731316431343662
+39656264323965653738316264636361343361303466326336663438643430366235643165323632
+34313331353530386266

ansible/host_vars/jar_lid_staging_local/main.yml

@@ -0,0 +1,28 @@
+---
+# Jar Lid Archive STAGING. Secrets are in vault.yml beside this file.
+#
+# These live here rather than in the role defaults because three plays read
+# them: the deploy on this box, the Caddy vhost on the edge host (port, domain)
+# and the backup (retention). A role default cannot reach a second play.
+
+# Names this deployment everywhere it must be unique across hosts: the Caddy site
+# file, the Gatus probe (probe_<instance>), the backup artefacts and the
+# backup-dump/backup-store endpoints.
+jar_lid_archive_instance: jar-lid-archive-staging
+
+# Fixed by the image's CMD (gunicorn --bind 0.0.0.0:8000); changing this moves
+# what Caddy and the probe call, not where gunicorn listens.
+jar_lid_archive_port: 8000
+jar_lid_archive_domain: "{{ subdomains.jar_lid_archive_staging }}.{{ root_domain }}"
+
+# Staging data is disposable: three days locally and on small-backups-box.
+jar_lid_archive_backup_retention_days: 3
+
+jar_lid_archive_seed_users:
+  - {username: alice, email: alice@example.com, password: "{{ jar_lid_archive_seed_password_alice }}"}
+  - {username: bob,   email: bob@example.com,   password: "{{ jar_lid_archive_seed_password_bob }}"}
+  - {username: admin, email: admin@example.com, password: "{{ jar_lid_archive_seed_password_admin }}", is_superuser: true}
+
+# No systemd units of ours run here - the stack is docker compose - so infra/401
+# has nothing to watch. The probe in the deploy playbook calls /health/, which
+# checks the database and the bucket as well as gunicorn.

ansible/host_vars/jar_lid_staging_local/vault.yml

@@ -0,0 +1,34 @@
+$ANSIBLE_VAULT;1.1;AES256
+37383535306133363031316535656461613038653562373839363432623163356234303061643566
+6533333535613064653336313664643063326562666639640a373530386539316363346332663137
+31303230323065623733663666383039346337343239646465393166663239643836356639373765
+3564313065363266360a643239333330616463616533623435656164386264303634356134303831
+36356131323637363734316161383065653663623732386638363165633265356362386538643039
+64636464323730316638316430643339356662353736366332663738656566663137366166393735
+34633430363931323330393234336661383532653366633735313365363562396137356438353330
+65313737313131326361326631363864396536343633623963303632623465363964373964393430
+61323161663939346435386435616364653336646165356232346335353235356339656336306138
+65613636353636623562613362633761333637613066663838396638613161663532393030303035
+35316464356539666631636265646433353537396564366538643962373364383730626164316265
+63393034663835346362636564316135326430633963393139646365333666663630336462323361
+31353830633335366635366435643864313665366563396134313938666537663232343162383366
+34636231636261663739383366363962366438323962393732393864623238343961363035333866
+61386630396334626564306536336561333265383464623730386263373132613438396332356631
+30386435333739653637633032313966626437303634613838653735373666343565306663376230
+61303734623865353835626464613033346437623234626136353664303434386262346634356463
+34396433353431393935386136333331643539633138636135353339323930386634393639666665
+36643462653061363062383039313936303062393364353366623965313530303031306336396336
+65643663343163636338393666386134366337633064376530303463393737396165663930383562
+64316363663636383764656139366433393766613262613433623637393538633130616232373136
+31376636633666626365376533356262393065656539613266326465643663383439373661393631
+30636136363066653936653837376634356333353036633165316662383337636435323162316430
+30316331353361396132653233623034376539323064363135306262376638353664643163653334
+37643638323038663531376336313064383238366430663363353733356530393661646138303230
+30633862626166653362346632626261636531373130353034656532386230616330643134383631
+63396138326336626431656130306531626539323934386332306361396239376466623866613364
+36666632316331326361373136326465323365653034363337313135656262623465306439646362
+30636639663163643430363165316231366232623332393635623461306339313762323265623732
+30633464343235323566356434376535313238663533633630356163336532303765633366316166
+38633462303762323930393337306265393033653061616337646163333335333737666332666632
+35613132646636333335313263343136323337616662623465653939336637653134383766336265
+35633638663636353537333238643236623033636163333164663062353863643164

ansible/host_vars/monerod_box_local/main.yml

@@ -12,11 +12,11 @@ monero_restricted_rpc_port: 18089
 # The explorer's HTTP port, read by the Caddy play on the edge host.
 xmrblocks_port: 8081
 
-# The 500 GB disk added to this VM in Proxmox as scsi1. Named here rather than
-# left to the role default so that "which disk holds the blockchain" is a
-# property of the machine, recorded next to the machine. The role only ever
-# uses it to create the filesystem; the mount itself goes by UUID.
-monero_data_device: /dev/sdb
+# The 500 GB chain disk, attached in Proxmox as scsi1. Named by that slot, which
+# is fixed when the disk is attached and survives reboots and storage moves -
+# not /dev/sdb, which it stopped being on 2026-10-04. Only read when the disk has
+# no monero-data filesystem yet; see roles/monero/defaults/main.yml.
+monero_data_device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
 
 # Systemd services deployed on this host, monitored every 5 minutes.
 #
@@ -29,24 +29,3 @@ monero_data_device: /dev/sdb
 monitored_services:
   - monerod
   - xmrblocks
-
-# ...but on THIS host, once a day rather than every five minutes.
-#
-# Both units are a Monero node in the middle of a multi-week initial sync on a
-# rotating disk. There is nothing a 5-minute poll reports that a daily one does
-# not, and every failing poll was another Signal message - 83 of them in six
-# hours while the explorer was wedged.
-#
-# This is safe to scope to the host because monerod-box runs ONLY Monero units.
-# Do not copy it to a box with a mixed workload: the timer is one per host and
-# iterates every unit in monitored_services, so slowing it slows all of them.
-#
-# 26h, not 24h: the window must EXCEED the push interval, or a check reporting
-# exactly on schedule races its own deadline and alerts on jitter alone. The
-# 900s randomized delay puts the real worst case at ~24h15m.
-#
-# Host liveness and disk are deliberately NOT slowed - they answer "is the box
-# alive", which is still wanted within minutes. See infra/400.
-service_health_on_calendar: "*-*-* 05:15:00"
-service_health_randomized_delay: "900"
-service_health_heartbeat: "26h"

ansible/host_vars/monitoring/main.yml

@@ -1,12 +0,0 @@
----
-
-# Systemd services deployed on this host, monitored every 5 minutes.
-#
-# The fact lives with the machine rather than in a central map, for the same
-# reason the cross-host ports do: "what runs here" is a property of the host,
-# and a central list is one more thing to forget to update when a service moves.
-#
-# Only units WE deploy belong here. Distro units (ssh, cron) have their own
-# supervision and would be noise.
-monitored_services:
-  - caddy

ansible/host_vars/spacey/main.yml

@@ -11,3 +11,10 @@
 monitored_services:
   - headscale
   - caddy
+
+# Gatus moved onto this host in the 2026-10-17 rebuild (the `monitoring` droplet
+# is gone). Headscale already listens on 0.0.0.0:8080, so Gatus takes 8081.
+# Host vars rather than role defaults: the Caddy vhost play in
+# services/gatus/deploy_gatus_playbook.yml does not load the role and reads
+# this same variable.
+gatus_port: 8081

ansible/host_vars/vipy/main.yml

@@ -13,3 +13,8 @@ monitored_services:
   - lnbits
   - caddy
   - phoenixd
+  # Deployed by the arbret repo, not this one (see arbret_handoff.md); listed
+  # here because "what runs on this host" is still a property of the host.
+  - arbret-server
+  - arbret-worker
+  - postgresql

ansible/infra/401_service_monitoring.yml

@@ -44,6 +44,13 @@
 #
 # Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints
 # have no built-in tolerance, so the threshold is where it belongs for them.
+#
+# EXCEPT here, where the threshold is 2. The heartbeat only forgives a MISSING
+# push. These checks also push an explicit `false` whenever a unit is caught
+# mid-restart - a deploy, an unattended upgrade, a backup stopping the service
+# for its dump - and at 1 every one of those pages, then resolves ten minutes
+# later. 2 means "down on two consecutive 5-minute runs", i.e. 5-10 minutes.
+# A host that stops reporting altogether still alarms on the heartbeat alone.
 # ─────────────────────────────────────────────────────────────────────────────
 - name: Register the service checks with Gatus
   hosts: observability
@@ -73,8 +80,9 @@
       vars:
         gatus_endpoint_default_alerts:
           - type: signal
-            # 1, not 3: the heartbeat window is the tolerance. See the note above.
-            failure-threshold: 1
+            # 2: tolerate one explicit `false` from a unit caught mid-restart.
+            # See the note above.
+            failure-threshold: 2
             success-threshold: 2
             send-on-resolved: true
             minimum-reminder-interval: 6h
@@ -99,9 +107,9 @@
         healthcheck_units: "{{ monitored_services }}"
         healthcheck_units_key_prefix: "services_{{ host_key }}"
         # Cadence and heartbeat window are ONE decision, and both are a property
-        # of the host - monerod-box runs a multi-week sync nobody needs polled
-        # every five minutes, so it declares a daily schedule in its host_vars
-        # and a matching window above. Everything else keeps 5min/11m.
+        # of the host: a host can set service_health_on_calendar and a matching
+        # service_health_heartbeat in its host_vars. None does today; everything
+        # runs 5min/11m. (monerod-box was daily during its initial sync.)
         #
         # An empty healthcheck_on_calendar means "use healthcheck_interval" -
         # see the timer template in roles/healthcheck.

ansible/infra/402_public_monitoring.yml

@@ -5,7 +5,8 @@
 # is the right way round for them: all three are about how the outside world
 # sees us, so they must be measured from outside. Gatus polls from the
 # observability host and needs nothing installed anywhere else - there is no
-# script, no timer and no token, because nothing is reporting in.
+# script, no timer and no push token, because nothing is reporting in. (The
+# provider-balance check does carry an API credential - see below.)
 #
 # That also means these have no heartbeat. A heartbeat answers "did the thing
 # that was supposed to report in do so"; when Gatus does the checking itself,
@@ -21,22 +22,24 @@
     # then left behind when the machine moved, so the check asserts against
     # ansible_host - if a box is renumbered, inventory is the one edit.
     dns_records:
-      - {sub: "{{ subdomains.gatus }}",              host: monitoring}
+      - {sub: "{{ subdomains.gatus }}",              host: spacey}
       - {sub: "{{ subdomains.headscale }}",          host: spacey}
       - {sub: "{{ subdomains.vaultwarden }}",        host: vipy}
       - {sub: "{{ subdomains.forgejo }}",            host: vipy}
       - {sub: "{{ subdomains.lnbits }}",             host: vipy}
-      - {sub: "{{ subdomains.ntfy_emergency_app }}", host: vipy}
       - {sub: "{{ subdomains.personal_blog }}",      host: vipy}
       - {sub: "{{ subdomains.memos }}",              host: vipy}
       - {sub: "{{ subdomains.mempool }}",            host: vipy}
       - {sub: "{{ subdomains.datum_gateway }}",      host: vipy}
-      # Daily rather than the 6h default - see the Monero note under
-      # public_sites. A record that has not moved in a year does not need
-      # four looks a day while the node behind it is mid-sync.
-      - {sub: "{{ subdomains.monero }}",             host: vipy, interval: '24h'}
-      - {sub: "{{ subdomains.monero_explorer }}",    host: vipy, interval: '24h'}
+      - {sub: "{{ subdomains.monero }}",             host: vipy}
+      - {sub: "{{ subdomains.monero_explorer }}",    host: vipy}
       - {sub: "{{ subdomains.music }}",              host: vipy}
+      - {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy}
+      # Not under root_domain, so named in full. arbret.com's zone is NOT in
+      # Porkbun (mynymdns) - this is the check that would have caught it still
+      # pointing at prd-arbret's recycled IP after the 2026-10-17 rebuild.
+      - {fqdn: arbret.com,     host: vipy}
+      - {fqdn: www.arbret.com, host: vipy}
 
     # A public resolver on purpose: this must test what the internet sees, not
     # what a local cache or the tailnet's MagicDNS happens to answer.
@@ -51,7 +54,6 @@
       - {name: vaultwarden, sub: "{{ subdomains.vaultwarden }}",        path: "/",       status: 200}
       - {name: forgejo,     sub: "{{ subdomains.forgejo }}",            path: "/",       status: 200}
       - {name: lnbits,      sub: "{{ subdomains.lnbits }}",             path: "/",       status: 200}
-      - {name: avisame,     sub: "{{ subdomains.ntfy_emergency_app }}", path: "/",       status: 200}
       - {name: blog,        sub: "{{ subdomains.personal_blog }}",      path: "/",       status: 200}
       - {name: memos,       sub: "{{ subdomains.memos }}",              path: "/",       status: 200}
       - {name: mempool,     sub: "{{ subdomains.mempool }}",            path: "/",       status: 200}
@@ -59,22 +61,25 @@
       # 401 is CORRECT: monerod answers an unauthenticated request with an HTTP
       # Digest challenge. Asserting 200 here would go green precisely when
       # rpc-login broke and the node started serving the world for free.
-      #
-      # Both Monero sites are polled ONCE A DAY, not every 5 minutes. The node
-      # is a multi-week initial sync on a rotating disk and the explorer reads
-      # the same LMDB, so a hard poll measures the disk more than the service -
-      # and a wedged explorer turned that cadence into 83 Signal messages in
-      # six hours. `threshold` drops with it: 3 failures at 5m is 15 minutes of
-      # tolerance, but at 24h it would be THREE DAYS before anyone heard. A
-      # daily check has no room for a blip, so it alerts on the first one.
-      - {name: monero,      sub: "{{ subdomains.monero }}",             path: "/get_info", status: 401, interval: '24h', threshold: 1}
-      - {name: exmrplore,   sub: "{{ subdomains.monero_explorer }}",    path: "/",       status: 200, interval: '24h', threshold: 1}
+      - {name: monero,      sub: "{{ subdomains.monero }}",             path: "/get_info", status: 401}
+      - {name: exmrplore,   sub: "{{ subdomains.monero_explorer }}",    path: "/",       status: 200}
       # /app/, not "/": checked live, "/" answers 302 (Navidrome redirects to
       # the SPA) and only /app/ returns 200 with the real UI. 200 rather than
       # 401 because Navidrome has no basic auth in front of it - the Subsonic
       # API the mobile clients speak cannot carry a proxy credential, so its
       # own login is the gate. See roles/navidrome/README.md.
       - {name: music,       sub: "{{ subdomains.music }}",              path: "/app/",   status: 200}
+      # /health/ through the whole public path: DNS, vipy's Caddy and cert, the
+      # tailnet hop, gunicorn, and the database and bucket behind it (503 if
+      # either is down). "/" would only prove the login redirect works.
+      - {name: jars-staging, sub: "{{ subdomains.jar_lid_archive_staging }}", path: "/health/", status: 200}
+      # arbret (runs on vipy since 2026-10-17; deployed by the arbret repo, not
+      # this one). /api/health answers 200 + "status":"ok" when healthy and 503
+      # "degraded" otherwise, DB down included - per the arbret owner. The body
+      # condition guards against a 200 that is not actually healthy.
+      - {name: arbret,        fqdn: arbret.com, path: "/",           status: 200}
+      - {name: arbret-health, fqdn: arbret.com, path: "/api/health", status: 200,
+         conditions: ['[BODY].status == ok']}
 
     # Ports published from the edge host by socket_proxy.
     public_tcp:
@@ -114,11 +119,11 @@
     - name: Build the DNS endpoints
       ansible.builtin.set_fact:
         dns_endpoints: "{{ dns_endpoints | default([]) + [{
-          'name': item.sub ~ '.' ~ root_domain,
+          'name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain),
           'group': 'dns',
           'url': dns_resolver,
           'interval': item.interval | default('6h'),
-          'dns': {'query-type': 'A', 'query-name': item.sub ~ '.' ~ root_domain},
+          'dns': {'query-type': 'A', 'query-name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain)},
           'conditions': ['[DNS_RCODE] == NOERROR',
                          '[BODY] == ' ~ hostvars[item.host].ansible_host],
           'alerts': [{'type': 'signal', 'failure-threshold': 1,
@@ -135,10 +140,11 @@
         http_endpoints: "{{ http_endpoints | default([]) + [{
           'name': item.name,
           'group': 'public',
-          'url': 'https://' ~ item.sub ~ '.' ~ root_domain ~ item.path,
+          'url': 'https://' ~ (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain) ~ item.path,
           'interval': item.interval | default('5m'),
           'conditions': ['[STATUS] == ' ~ item.status,
-                         '[CERTIFICATE_EXPIRATION] > 168h'],
+                         '[CERTIFICATE_EXPIRATION] > 168h']
+                        + item.conditions | default([]),
           'alerts': [{'type': 'signal', 'failure-threshold': item.threshold | default(3),
                       'success-threshold': 2, 'send-on-resolved': true,
                       'minimum-reminder-interval': '6h'}]}] }}"
@@ -158,9 +164,50 @@
                       'minimum-reminder-interval': '6h'}]}] }}"
       loop: "{{ public_tcp }}"
 
+    # ── Provider balance ─────────────────────────────────────────────────────
+    # 99stack is prepaid: when the balance runs out the servers stop. The one
+    # pulled check here that carries a credential - unlike everything above,
+    # the API will not answer anonymously (403 "Authorization token is
+    # missing").
+    #
+    # The condition is RUNWAY, not balance. billing.secondsdue is the
+    # provider's own "seconds until you run out at current usage", so it
+    # follows the spend as servers come and go; a fixed balance threshold
+    # would be one more number written down once and left behind. 336h of
+    # runway, same as domain expiry, because topping up is a manual act too.
+    #
+    # The token is account-wide - 99stack has no read-only scope - so it can
+    # create and destroy servers. It lands in Gatus's endpoints file (0640
+    # root:gatus), the same place as the push tokens.
+    - name: Require the 99stack API token
+      ansible.builtin.assert:
+        that:
+          - ninetyninestack_api_token is defined
+          - ninetyninestack_api_token | length > 0
+        fail_msg: >-
+          ninetyninestack_api_token must be in group_vars/all/vault.yml - an
+          API token from the 99stack control panel.
+        quiet: true
+
+    - name: Build the provider endpoints
+      ansible.builtin.set_fact:
+        provider_endpoints:
+          - name: 99stack-balance
+            group: provider
+            url: https://api.99stack.com/v1.5/account/info
+            interval: 6h
+            headers:
+              Authorization: "Bearer {{ ninetyninestack_api_token }}"
+            conditions:
+              - "[STATUS] == 200"
+              - "[BODY].billing.secondsdue > {{ 14 * 86400 }}"
+            alerts:
+              - {type: signal, failure-threshold: 1, success-threshold: 1,
+                 send-on-resolved: true, minimum-reminder-interval: 24h}
+
     - name: Register the public-facing endpoints
       ansible.builtin.include_role:
         name: gatus_endpoint
       vars:
         gatus_endpoint_name: public
-        gatus_endpoint_pulled: "{{ domain_endpoints + dns_endpoints + http_endpoints + tcp_endpoints }}"
+        gatus_endpoint_pulled: "{{ domain_endpoints + dns_endpoints + http_endpoints + tcp_endpoints + provider_endpoints }}"

ansible/infra/403_service_probe_registration.yml

@@ -37,7 +37,13 @@
 
   vars:
     probes:
-      - {name: bitcoin-knots,    host: knots_box_local}
+      # failure_threshold 4: this probe runs every MINUTE, and bitcoind turns RPC
+      # away for 1-3 minutes when a block lands - validation on the saturated
+      # HDD plus Fulcrum/mempool/DATUM all fetching the block at once (93
+      # "work queue depth exceeded" in one day, 2026-10-18). At 1, every slow
+      # block paged. 4 consecutive failed minutes is past any block burst seen,
+      # and a real outage still alerts within ~5 minutes.
+      - {name: bitcoin-knots,    host: knots_box_local, failure_threshold: 4}
       - {name: datum-gateway,    host: knots_box_local}
       - {name: fulcrum,          host: fulcrum_box_local}
       - {name: phoenixd,         host: vipy}
@@ -45,16 +51,33 @@
       - {name: mempool-mariadb,  host: mempool_box_local}
       - {name: mempool-backend,  host: mempool_box_local}
       - {name: mempool-frontend, host: mempool_box_local}
-      # The two Monero probes push ONCE A DAY, not every 5 minutes, so they
-      # need a window wider than a day. 26h, not 24h: the window must exceed
-      # the push interval or a check that reports exactly on time races its own
-      # deadline and alerts on jitter. The timers are OnCalendar 05:00 and
-      # 05:30 with RandomizedDelaySec=900, so the true worst case is ~24h15m.
-      # Cadence lives in roles/{monero,xmrblocks}/defaults/main.yml; the two
-      # are one decision and must move together.
-      - {name: monerod,          host: monerod_box_local, heartbeat: '26h'}
-      - {name: xmrblocks,        host: monerod_box_local, heartbeat: '26h'}
+      # Every 5 minutes like the rest since 2026-10-17 (they were daily while
+      # the node synced on a rotating disk). Cadence lives in
+      # roles/{monero,xmrblocks}/defaults/main.yml; it and this window are one
+      # decision and must move together.
+      - {name: monerod,          host: monerod_box_local}
+      - {name: xmrblocks,        host: monerod_box_local}
       - {name: navidrome,        host: music_box_local}
+      - {name: jar-lid-archive-staging, host: jar_lid_staging_local}
+      # arbret (on vipy, deployed by the arbret repo). `host` here is the KEY in
+      # gatus_push_tokens, and arbret has a dedicated `arbret` token rather than
+      # vipy's - so it cannot write vipy's own results, and vice versa.
+      # Windows are two runs plus slack, as for the 5m probes above:
+      #   analytics  hourly at :00       -> 130m
+      #   beancount  daily at 03:00      -> 26h
+      # Cadence lives in the arbret repo; tell us if it changes.
+      - {name: arbret-analytics, host: arbret, heartbeat: '130m'}
+      - {name: arbret-beancount, host: arbret, heartbeat: '26h'}
+
+    # The alert every probe gets. failure-threshold is 1 - the heartbeat window
+    # is the tolerance, see the note above - unless a probe sets
+    # failure_threshold, for checks that legitimately fail for a minute or two.
+    probe_alert:
+      type: signal
+      failure-threshold: 1
+      success-threshold: 2
+      send-on-resolved: true
+      minimum-reminder-interval: 6h
 
   tasks:
     - name: Build the probe endpoint list
@@ -63,19 +86,13 @@
           'name': item.name,
           'group': 'probe',
           'token': gatus_push_tokens[item.host],
-          'heartbeat': item.heartbeat | default('11m')}] }}"
+          'heartbeat': item.heartbeat | default('11m'),
+          'alerts': [probe_alert | combine({'failure-threshold': item.failure_threshold | default(1)})]}] }}"
       loop: "{{ probes }}"
 
     - name: Register the probe endpoints
       ansible.builtin.include_role:
         name: gatus_endpoint
       vars:
-        gatus_endpoint_default_alerts:
-          - type: signal
-            # 1, not 3: the heartbeat window is the tolerance. See the note above.
-            failure-threshold: 1
-            success-threshold: 2
-            send-on-resolved: true
-            minimum-reminder-interval: 6h
         gatus_endpoint_name: probes
         gatus_endpoint_external: "{{ probe_endpoints }}"

ansible/inventory.ini

@@ -1,7 +1,6 @@
 [vps]
-vipy ansible_host=167.172.107.33 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
-spacey ansible_host=64.227.112.128 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
-monitoring ansible_host=64.226.70.190 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+vipy ansible_host=167.172.98.204 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+spacey ansible_host=46.101.250.26 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 
 [nodito_host]
 nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
@@ -11,9 +10,9 @@ nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ans
 knots_box_local ansible_host=knots-box lan_ip=192.168.1.135 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 fulcrum_box_local ansible_host=fulcrum-box lan_ip=192.168.1.140 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 mempool_box_local ansible_host=mempool-box lan_ip=192.168.1.142 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
-memos_box_local ansible_host=memos-box lan_ip=192.168.1.145 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+memos_box_local ansible_host=memos-box lan_ip=192.168.1.130 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 forgejo_runner_local ansible_host=forgejo-runner-box lan_ip=192.168.1.132 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
-arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.147 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.145 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 small_backups_local ansible_host=small-backups-box lan_ip=192.168.1.131 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 # lan_ip read off nodito's ARP table for the VM's own MAC (bc:24:11:89:fc:1f),
@@ -21,23 +20,22 @@ monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=cou
 # key was still in known_hosts and SSH refused the connection, which is the
 # designed behaviour described in group_vars/nodito_vms.yml.
 music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+# lan_ip read from the Proxmox guest agent for VM 3300 (qm guest cmd
+# network-get-interfaces), and the host key checked against the one the agent
+# reads off the disk before it went into known_hosts.
+jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1.151 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 
 # Local connection to laptop: this assumes you're running ansible commands from your personal laptop
 [lapy]
 localhost ansible_connection=local ansible_user=counterweight
 
-[arbret]
-prd-arbret ansible_host=167.99.242.62 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
-
 [edge]
 vipy
 
-# The group is `observability`, NOT `monitoring` — there is a HOST named
-# `monitoring` on line 5, and a group with the same name makes `hosts: monitoring`
-# ambiguous. Ansible resolved it to the host and warned:
-#   [WARNING]: Found both group and host with same name: monitoring
+# Gatus and its Signal transport. Shares spacey with headscale since the
+# 2026-10-17 rebuild; the separate `monitoring` droplet is gone.
 [observability]
-monitoring
+spacey
 
 [vpn_control]
 spacey
@@ -63,6 +61,10 @@ memos_box_local
 [music]
 music_box_local
 
+# Staging only for now; a production host joins this group with its own host_vars.
+[jar_lid_archive]
+jar_lid_staging_local
+
 [ci_runner]
 forgejo_runner_local
 
@@ -70,7 +72,7 @@ forgejo_runner_local
 localhost
 
 # Every machine Ansible may configure as a server.
-# Deliberately EXCLUDES [control] (your laptop) and [arbret].
+# Deliberately EXCLUDES [control] (your laptop).
 [managed:children]
 vps
 nodito_host

ansible/playbooks/backups.yml

@@ -12,8 +12,10 @@
         backup_store_check_push_base: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints"
         backup_store_check_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
         backup_store_sources:
+          # On vipy since 2026-10-17 (prd-arbret is gone). The arbret repo writes
+          # the dump to the backup_source contract - see arbret_handoff.md.
           - name: arbret
-            source: "arbret@prd-arbret:/opt/arbret/backups/"
+            source: "backup-pull@prd-vipy:/opt/backups/arbret/"
             retention_days: 90
           - name: headscale
             source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/"
@@ -30,6 +32,10 @@
           - name: forgejo
             source: "backup-pull@prd-vipy:/opt/backups/forgejo/"
             retention_days: 14
+          # Staging: disposable data, so the long tail is as short as the local one.
+          - name: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}"
+            source: "backup-pull@{{ hostvars['jar_lid_staging_local'].ansible_host }}:/opt/backups/{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}/"
+            retention_days: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_backup_retention_days }}"
 
 # ─────────────────────────────────────────────────────────────────────────────
 # Register the backup checks with Gatus.
@@ -48,8 +54,8 @@
 # LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it
 # cannot tell you whether the dump broke or the pull did.
 #
-# arbret has no dump endpoint: prd-arbret lives in [arbret], which `managed`
-# deliberately excludes, so nothing of ours runs there. It is store-checked only.
+# arbret's dump is written by the arbret repo, not by backup_source, and it
+# pushes with its own `arbret` token (the `host` below is the token key).
 # ─────────────────────────────────────────────────────────────────────────────
 - name: Register the backup checks with Gatus
   hosts: observability
@@ -62,7 +68,9 @@
       - {name: vaultwarden, host: vipy}
       - {name: lnbits,      host: vipy}
       - {name: forgejo,     host: vipy}
-    store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo]
+      - {name: jar-lid-archive-staging, host: jar_lid_staging_local}
+      - {name: arbret,      host: arbret}
+    store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo, jar-lid-archive-staging]
 
   tasks:
     # 26h, not 7h: the DUMP is genuinely daily, so the window cannot be tighter

ansible/roles/bitcoin_knots/defaults/main.yml

@@ -22,6 +22,15 @@ bitcoin_rpc_port: 8332
 bitcoin_p2p_port: 8333
 bitcoin_rpc_bind: "0.0.0.0"
 
+# RPC capacity. When a block lands, Fulcrum, mempool and DATUM all fetch it at
+# once while validation holds the node busy on a saturated HDD; with the
+# defaults the queue overflowed and bitcoind REJECTED calls ("http work queue
+# depth exceeded" - 93 times on 2026-10-18), which also made the health probe
+# flap. A deeper queue lets those bursts wait instead of failing. Costs a few
+# idle threads and a little memory.
+bitcoin_rpcworkqueue: 64
+bitcoin_rpcthreads: 8
+
 # Build options
 bitcoin_build_jobs: 4  # Parallel build jobs (-j flag), adjust based on CPU cores
 bitcoin_build_prefix: /usr/local
@@ -60,11 +69,17 @@ healthcheck_push_token: ""
 bitcoin_logfile: "{{ bitcoin_data_dir }}/debug.log"
 
 # --- dbcache ----------------------------------------------------------------
-# Computed as 90% of RAM unless this is set. The live node was hand-tuned to
-# 200 MB; the calculation would have produced 3528. As with fulcrum, note that
+# Computed as 90% of RAM unless this is set. As with fulcrum, note that
 # set_fact outranks role defaults, so the CALCULATION has to honour this - a
 # value pinned only in defaults/ is silently ignored.
-bitcoin_dbcache_mb_override: 200
+#
+# 1200, raised from a hand-tuned 200 on 2026-10-18. At 200 MB almost no UTXO
+# lookup hit memory: the P2P thread alone read ~4 MB/s off the saturated HDD
+# mirror validating relayed transactions, and block validation took a median
+# 11 s, p90 103 s, max 344 s - long enough for RPC to stall and the health
+# probe to fail. 1200 is what fits in the 4 GB VM (bitcoind ~1.5 GB RSS at
+# 200, mempool ~3 MB in practice). Raising the VM to 6 GB would allow ~3000.
+bitcoin_dbcache_mb_override: 1200
 
 # --- DATUM Gateway ----------------------------------------------------------
 # This node feeds block templates to datum_gateway on knots-box. These settings

ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2

@@ -11,6 +11,8 @@ rpcpassword={{ bitcoin_rpc_password }}
 rpcbind={{ bitcoin_rpc_bind }}
 rpcport={{ bitcoin_rpc_port }}
 rpcallowip=0.0.0.0/0
+rpcworkqueue={{ bitcoin_rpcworkqueue }}
+rpcthreads={{ bitcoin_rpcthreads }}
 
 # Network Configuration
 listen=1

ansible/roles/gatus_endpoint/defaults/main.yml

@@ -12,6 +12,7 @@ gatus_endpoint_name: ""
 #
 # A DNS check adds `dns: {query-type, query-name}` - and note that for those,
 # `url` is the RESOLVER to ask, not the name being looked up.
+# An API behind a credential adds `headers: {Authorization: ...}`.
 # A domain-expiry check is just `url: <domain>` with a [DOMAIN_EXPIRATION]
 # condition; it uses WHOIS/RDAP and needs no scheme.
 gatus_endpoint_pulled: []

ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2

@@ -41,6 +41,10 @@ endpoints:
 {% if e.client is defined %}
     client:
 {{ e.client | to_nice_yaml(indent=2) | indent(6, true) }}
+{% endif %}
+{% if e.headers is defined %}
+    headers:
+{{ e.headers | to_nice_yaml(indent=2) | indent(6, true) }}
 {% endif %}
     conditions:
 {% for c in e.conditions %}

ansible/roles/jar_lid_archive/README.md

@@ -0,0 +1,121 @@
+# `jar_lid_archive`
+
+Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive)
+as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and
+SeaweedFS as the S3 photo store (`storage`). The same three services as the app
+repo's own `compose.yaml`, so staging rehearses what dev and CI run.
+
+Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to
+every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local`
+(jar-lid-archive-staging-box, nodito VM 3300), public at
+`jars-staging.contrapeso.xyz` through vipy's Caddy.
+
+Per-host values are in `host_vars/<host>/main.yml`, and secrets are in `vault.yml`
+beside it. A production host would be a second inventory entry with its own
+`jar_lid_archive_instance`, domain and vault, not a second playbook.
+
+## Updates: follows `:master`
+
+CI pushes `:master` on every green build of master. Every run of the playbook
+runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only
+if the digest moved. There is no watchtower, so **a merge reaches staging when
+the playbook runs**, not on its own. The run ends by printing `/health/`, which
+includes the deployed version.
+
+`up --wait` only returns once every container's healthcheck passes. `web`'s is
+`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means
+the app actually serves.
+
+## Networking: why `web` is on the host network
+
+Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT`
+rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well
+as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids
+that, but at boot Docker races tailscaled for the address and loses.
+
+So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host
+port:
+
+- **LAN:** UFW default-deny blocks it.
+- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW
+  (checked on mempool-box), so the headscale ACL is the real control. It grants
+  `tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`.
+- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches
+  them from the host network.
+
+This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto`
+(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only
+thing that can reach gunicorn. Caddy overwrites whatever value the client sent.
+
+## Secrets and seed users
+
+The Django secret key, the Postgres password, the S3 secret and the three seed
+passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They
+reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose
+interpolates. `docker-compose.yml` itself has no secrets.
+
+The seed users are alice, bob and admin (superuser), with the fixture's usernames
+and **vault** passwords. The fixture's `*-test-pass` passwords are published in
+the app repo and this site is public. `seed_users` runs on every deploy and
+resets each listed user's password, so rotating one means editing the vault and
+re-running the playbook.
+
+```sh
+ansible-vault view host_vars/jar_lid_staging_local/vault.yml   # from ansible/
+```
+
+## Backups
+
+`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses
+`backup_source`. At 02:20 it stops `web` only, and
+`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with:
+
+- `db.dump`: `pg_dump -Fc`
+- `seaweedfs/`: the photo store's data directory, as files
+
+`web` is the only writer to the database and the photo store. With it stopped,
+both are quiet and are captured at the same instant. `storage` is deliberately
+left running: on this disk it can take minutes to restart, and the first test run
+that stopped it left the app down for 26 minutes. Uploads fail while `web` is
+down.
+
+The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally,
+and small-backups-box keeps three as well. Staging data is disposable.
+
+### Restore
+
+```sh
+age -d -i ~/.age/counterweight_age jar-lid-archive-staging_<ts>.tar.age | tar -xf - -C /tmp/restore
+# on the box, stack down:
+docker compose -f /opt/jar-lid-archive/docker-compose.yml down
+rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/
+docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db
+docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \
+  pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump
+docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait
+```
+
+## Monitoring
+
+| What | Where |
+|---|---|
+| Host liveness and disk | infra/400 (automatic for `managed`) |
+| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) |
+| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) |
+| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` |
+
+infra/401 has nothing to watch here, because no systemd units of ours run on
+this box.
+
+## Size and disk
+
+The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's
+random reads keep that mirror close to saturated, and this box has measured
+**~96% "full" I/O pressure** with nothing of its own running. Everything is slow
+here, but especially container starts. The healthcheck `start_period`s are sized
+for that. Moving the disk to `local-zfs` (NVMe) would fix it.
+
+
+1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian,
+the images, the photos and three days of local backups, and each backup is
+about the size of the whole photo store. infra/400's disk check alarms at 85%.

ansible/roles/jar_lid_archive/defaults/main.yml

@@ -0,0 +1,57 @@
+---
+# Jar Lid Archive: Django (gunicorn) + Postgres + SeaweedFS (S3) as one compose
+# stack. Source and CI live in forgejo.contrapeso.xyz/counterweight/jar-lid-archive.
+#
+# Per-deployment values (domain, port, retention) live in the host's host_vars,
+# because the Caddy play on the edge host and the backup plays read them too and
+# a role default cannot reach a second play.
+
+jar_lid_archive_dir: /opt/jar-lid-archive
+
+# ── Images ───────────────────────────────────────────────────────────────────
+# The app follows a moving tag on purpose: every run pulls it, and compose
+# recreates `web` only when the digest actually changed. CI pushes :master on
+# every green build of master, plus :<sha>.
+jar_lid_archive_image: forgejo.contrapeso.xyz/counterweight/jar-lid-archive
+jar_lid_archive_image_tag: master
+
+# Pinned to what the app's own compose.yaml and CI run, so staging rehearses the
+# same stack. Bump them together with the app repo, not separately.
+jar_lid_archive_postgres_image: "postgres:18"
+jar_lid_archive_seaweedfs_image: "chrislusf/seaweedfs:4.47"
+
+# Registry login. Empty = pull anonymously, which works today: the package is
+# owner-scoped in Forgejo and public even though the repository is private.
+jar_lid_archive_registry: forgejo.contrapeso.xyz
+jar_lid_archive_registry_user: ""
+jar_lid_archive_registry_token: ""
+
+# ── Networking ───────────────────────────────────────────────────────────────
+# gunicorn listens on the HOST network on this port. See the README for why host
+# networking rather than a published port: Docker's published ports bypass UFW.
+jar_lid_archive_port: 8000
+# Loopback-only ports for the two backing services, so the web container (on the
+# host network) can reach them and nothing else can.
+jar_lid_archive_postgres_port: 5432
+jar_lid_archive_s3_port: 8333
+
+jar_lid_archive_domain: ""              # e.g. jars-staging.contrapeso.xyz
+
+# ── App settings ─────────────────────────────────────────────────────────────
+jar_lid_archive_postgres_db: jarlid
+jar_lid_archive_postgres_user: jarlid
+jar_lid_archive_s3_access_key: jarlid
+jar_lid_archive_s3_bucket: photos
+
+# Secrets: pass from the vault. The assert in tasks/deploy.yml refuses empties.
+jar_lid_archive_django_secret_key: ""
+jar_lid_archive_postgres_password: ""
+jar_lid_archive_s3_secret_key: ""
+
+# ── Seed users ───────────────────────────────────────────────────────────────
+# Loaded with `manage.py seed_users -` on every run. That command RESETS each
+# listed user's password and flags, so the vault stays the source of truth.
+# Same usernames as the app's tests/fixtures/users.json, never its passwords:
+# this deployment is reachable from the internet.
+#   - {username: alice, password: "...", email: "...", is_superuser: false}
+jar_lid_archive_seed_users: []

ansible/roles/jar_lid_archive/tasks/deploy.yml

@@ -0,0 +1,100 @@
+---
+- name: Assert the deployment is fully specified
+  ansible.builtin.assert:
+    that:
+      - jar_lid_archive_domain | length > 0
+      - jar_lid_archive_django_secret_key | length >= 50
+      - jar_lid_archive_postgres_password | length > 0
+      - jar_lid_archive_s3_secret_key | length > 0
+    fail_msg: >-
+      jar_lid_archive needs a domain and its three secrets from the vault. The
+      app falls back to development defaults for every one of them, so an empty
+      value here would not fail - it would deploy with a published secret key.
+  no_log: true
+
+- name: Create the deployment directory
+  ansible.builtin.file:
+    path: "{{ jar_lid_archive_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: '0755'
+
+# Existence only, no owner: the postgres and seaweedfs entrypoints chown their
+# own data directories to the uid inside the container. Declaring an owner here
+# would fight them on every run - the mempool MariaDB lesson.
+- name: Ensure the data directories exist
+  ansible.builtin.file:
+    path: "{{ jar_lid_archive_dir }}/{{ item }}"
+    state: directory
+  loop:
+    - postgres
+    - seaweedfs
+
+# diff: false, because --diff prints rendered content and this is all secrets.
+- name: Write the compose environment file
+  ansible.builtin.template:
+    src: env.j2
+    dest: "{{ jar_lid_archive_dir }}/.env"
+    owner: root
+    group: root
+    mode: '0600'
+  diff: false
+
+- name: Write docker-compose.yml
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ jar_lid_archive_dir }}/docker-compose.yml"
+    owner: root
+    group: root
+    mode: '0644'
+
+- name: Log in to the container registry
+  ansible.builtin.command:
+    cmd: "docker login {{ jar_lid_archive_registry }} -u {{ jar_lid_archive_registry_user }} --password-stdin"
+    stdin: "{{ jar_lid_archive_registry_token }}"
+  when: jar_lid_archive_registry_token | length > 0
+  changed_when: false
+  no_log: true
+
+# Pulled on every run: staging follows :master. `up` below recreates `web` only
+# when the pulled digest differs from the running one.
+- name: Pull images
+  ansible.builtin.command: docker compose pull --quiet
+  args:
+    chdir: "{{ jar_lid_archive_dir }}"
+  changed_when: false
+
+# --wait blocks until every service's healthcheck passes, and `web`'s is
+# /health/, which itself checks the database and the bucket - so this task
+# succeeding means the app actually serves, not merely that containers exist.
+- name: Start the stack and wait for it to be healthy
+  ansible.builtin.command: docker compose up -d --remove-orphans --wait --wait-timeout 300
+  args:
+    chdir: "{{ jar_lid_archive_dir }}"
+  register: jar_lid_archive_up
+  changed_when: jar_lid_archive_up.stderr is search('Created|Recreated|Started|Removed')
+
+- name: Read the deployed version
+  ansible.builtin.uri:
+    url: "http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
+    return_content: true
+  register: jar_lid_archive_health
+  # Under --check the stack was never started, so there is nothing to ask.
+  when: not ansible_check_mode
+
+- name: Show the deployed version
+  ansible.builtin.debug:
+    msg: "{{ jar_lid_archive_health.json }}"
+  when: not ansible_check_mode
+
+# Used by the backup (services/jar-lid-archive/setup_backup_jar_lid_archive.yml).
+# Lives here because what to dump is a fact about this stack's layout.
+- name: Install the dump script
+  ansible.builtin.template:
+    src: dump.sh.j2
+    dest: /usr/local/bin/jar-lid-archive-dump.sh
+    owner: root
+    group: root
+    mode: '0755'
+    validate: "bash -n %s"

ansible/roles/jar_lid_archive/tasks/docker.yml

@@ -0,0 +1,74 @@
+---
+# Same Docker CE install as roles/mempool/tasks/docker.yml. infra/910_docker
+# targets every managed host, so each Docker role installs its own engine.
+- name: Remove old Docker-related packages
+  apt:
+    name:
+      - docker.io
+      - docker-doc
+      - docker-compose
+      - podman-docker
+      - containerd
+      - runc
+    state: absent
+    purge: yes
+    autoremove: yes
+
+- name: Update apt cache
+  apt:
+    update_cache: yes
+
+- name: Install prerequisites
+  apt:
+    name:
+      - ca-certificates
+      - curl
+    state: present
+
+- name: Create directory for Docker GPG key
+  file:
+    path: /etc/apt/keyrings
+    state: directory
+    mode: '0755'
+
+- name: Download Docker GPG key
+  get_url:
+    url: https://download.docker.com/linux/debian/gpg
+    dest: /etc/apt/keyrings/docker.asc
+    mode: '0644'
+
+- name: Get Debian architecture
+  command: dpkg --print-architecture
+  register: deb_arch
+  changed_when: false
+
+- name: Add Docker repository
+  apt_repository:
+    repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable"
+    filename: docker
+    state: present
+    update_cache: yes
+
+- name: Install Docker packages
+  apt:
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-buildx-plugin
+      - docker-compose-plugin
+    state: present
+    update_cache: yes
+
+- name: Ensure Docker is started and enabled
+  systemd:
+    name: docker
+    enabled: yes
+    state: started
+
+- name: Add user to docker group
+  user:
+    name: "{{ ansible_user }}"
+    groups: docker
+    append: yes
+

ansible/roles/jar_lid_archive/tasks/main.yml

@@ -0,0 +1,6 @@
+---
+# import_tasks, not include_tasks: static imports stay visible to --list-tasks,
+# which is the main verification tool in this repo.
+- ansible.builtin.import_tasks: docker.yml
+- ansible.builtin.import_tasks: deploy.yml
+- ansible.builtin.import_tasks: seed.yml

ansible/roles/jar_lid_archive/tasks/seed.yml

@@ -0,0 +1,13 @@
+---
+# seed_users is idempotent but RESETS each listed user's password and flags, so
+# re-running it is how a rotated vault password reaches the app. It reports
+# "N created, M updated"; only a creation counts as a change.
+- name: Seed users
+  ansible.builtin.command:
+    cmd: docker compose exec -T web python manage.py seed_users -
+    chdir: "{{ jar_lid_archive_dir }}"
+    stdin: "{{ jar_lid_archive_seed_users | to_json }}"
+  register: jar_lid_archive_seed
+  changed_when: "jar_lid_archive_seed.stdout is not search('Seeded users: 0 created')"
+  when: jar_lid_archive_seed_users | length > 0
+  no_log: true

ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2

@@ -0,0 +1,123 @@
+# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
+#
+# Same three services as the app repo's compose.yaml, with three differences:
+#   - `web` runs the CI-built image instead of `build: .`
+#   - `web` is on the HOST network, so UFW and the tailnet ACL decide who reaches
+#     gunicorn. A published port would be DNATed by Docker straight past UFW and
+#     answer on the LAN too. See the role README.
+#   - data is bind-mounted under {{ jar_lid_archive_dir }} rather than in named
+#     volumes, so the backup tars a path it can name.
+#
+# Secrets are interpolated from .env (0600) beside this file; this file has none.
+name: jar-lid-archive
+
+x-logging: &logging
+  # Docker's default json-file driver never rotates, on a 10 GB disk.
+  driver: json-file
+  options:
+    max-size: "10m"
+    max-file: "3"
+
+x-postgres-env: &postgres-env
+  POSTGRES_DB: "{{ jar_lid_archive_postgres_db }}"
+  POSTGRES_USER: "{{ jar_lid_archive_postgres_user }}"
+  POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?}
+
+x-s3-env: &s3-env
+  S3_ACCESS_KEY: "{{ jar_lid_archive_s3_access_key }}"
+  S3_SECRET_KEY: ${S3_SECRET_KEY:?}
+
+services:
+  web:
+    image: "{{ jar_lid_archive_image }}:{{ jar_lid_archive_image_tag }}"
+    # gunicorn binds 0.0.0.0:{{ jar_lid_archive_port }} - the port is fixed by the image's CMD.
+    network_mode: host
+    environment:
+      <<: [*postgres-env, *s3-env]
+      POSTGRES_HOST: 127.0.0.1
+      POSTGRES_PORT: "{{ jar_lid_archive_postgres_port }}"
+      S3_ENDPOINT_URL: "http://127.0.0.1:{{ jar_lid_archive_s3_port }}"
+      S3_BUCKET: "{{ jar_lid_archive_s3_bucket }}"
+      DJANGO_DEBUG: "0"
+      DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY:?}
+      # 127.0.0.1 for the container healthcheck and the probe, which both call
+      # /health/ directly rather than through Caddy.
+      DJANGO_ALLOWED_HOSTS: "{{ jar_lid_archive_domain }},127.0.0.1,localhost"
+      # Caddy terminates TLS and forwards plain HTTP with X-Forwarded-Proto. The
+      # app must trust that header or every POST fails CSRF and share links come
+      # out http://. Safe only because nothing but Caddy can reach gunicorn.
+      DJANGO_BEHIND_TLS_PROXY: "1"
+      DJANGO_CSRF_TRUSTED_ORIGINS: "https://{{ jar_lid_archive_domain }}"
+    depends_on:
+      db:
+        condition: service_healthy
+      storage:
+        condition: service_healthy
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:{{ jar_lid_archive_port }}/health/')"]
+      interval: 30s
+      timeout: 5s
+      retries: 5
+      start_period: 180s
+    logging: *logging
+    restart: unless-stopped
+
+  db:
+    image: "{{ jar_lid_archive_postgres_image }}"
+    environment: *postgres-env
+    ports:
+      - "127.0.0.1:{{ jar_lid_archive_postgres_port }}:5432"
+    volumes:
+      - ./postgres:/var/lib/postgresql
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
+      interval: 5s
+      timeout: 3s
+      retries: 15
+      start_period: 120s
+    logging: *logging
+    restart: unless-stopped
+
+  storage:
+    image: "{{ jar_lid_archive_seaweedfs_image }}"
+    environment: *s3-env
+    # Copied from the app's compose.yaml: write the S3 identity from the
+    # environment, then hand over to the image's entrypoint, which drops root
+    # and runs master + volume + filer + S3 gateway.
+    #
+    # The `rm -f` is ours, and load-bearing. On a RESTART of the same container
+    # (a reboot, a backup's stop/start) /tmp/s3.json survives, owned by seaweed
+    # from the chown below. Root cannot then re-create it: /tmp is sticky and
+    # the kernel's protected_regular refuses O_CREAT on another user's file
+    # there. The result was "can't create /tmp/s3.json: Permission denied", an
+    # unhealthy storage container, and web never starting - found on the first
+    # backup run.
+    entrypoint: ["sh", "-c"]
+    command:
+      - |
+        rm -f /tmp/s3.json
+        umask 077
+        cat > /tmp/s3.json <<EOF
+        {"identities": [{"name": "app",
+          "credentials": [{"accessKey": "$$S3_ACCESS_KEY", "secretKey": "$$S3_SECRET_KEY"}],
+          "actions": ["Admin", "Read", "Write", "List", "Tagging"]}]}
+        EOF
+        chown seaweed /tmp/s3.json
+        exec /entrypoint.sh server -s3 -s3.config=/tmp/s3.json
+    ports:
+      - "127.0.0.1:{{ jar_lid_archive_s3_port }}:8333"
+    volumes:
+      - ./seaweedfs:/data
+    # start_period, because this VM's disk is on nodito's HDD mirror, shared
+    # with knots-box. Under that contention SeaweedFS has taken over the 150s
+    # that retries x interval allowed; compose then marked it unhealthy, and
+    # web's depends_on gave up and never started it - a 26-minute outage.
+    # Failures inside start_period do not count.
+    healthcheck:
+      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8333/healthz"]
+      interval: 5s
+      timeout: 3s
+      retries: 30
+      start_period: 300s
+    logging: *logging
+    restart: unless-stopped

ansible/roles/jar_lid_archive/templates/dump.sh.j2

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Jar Lid Archive dump - managed by Ansible (roles/jar_lid_archive).
+#
+# Writes ONE uncompressed tar to STDOUT, for roles/backup_source to encrypt:
+#   db.dump      pg_dump custom format   -> pg_restore -d jarlid db.dump
+#   seaweedfs/   the photo store, as files
+#
+# Called with `web` stopped (the backup playbook arranges it). web is the only
+# writer to db and storage, so both are quiescent and agree with each other: no
+# row points at a photo the tar does not have. storage keeps running; with no
+# writes arriving its volume files do not change under tar.
+#
+# The pg_dump lands in a temp dir on the same disk as the live database, which
+# is already plaintext; it is gone before this script exits.
+#
+# tar, not tar.gz: the bulk is JPEGs, which do not compress, and gzip on one
+# core would only make the window longer.
+set -euo pipefail
+
+cd {{ jar_lid_archive_dir }}
+tmp=$(mktemp -d {{ jar_lid_archive_dir }}/.dump.XXXXXX)
+trap 'rm -rf "$tmp"' EXIT
+
+docker compose exec -T db pg_dump -U {{ jar_lid_archive_postgres_user }} -d {{ jar_lid_archive_postgres_db }} -Fc > "$tmp/db.dump"
+tar -cf - -C "$tmp" db.dump -C {{ jar_lid_archive_dir }} seaweedfs

ansible/roles/jar_lid_archive/templates/env.j2

@@ -0,0 +1,5 @@
+# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
+# Read by docker compose to interpolate docker-compose.yml. 0600: secrets.
+POSTGRES_PASSWORD={{ jar_lid_archive_postgres_password }}
+S3_SECRET_KEY={{ jar_lid_archive_s3_secret_key }}
+DJANGO_SECRET_KEY={{ jar_lid_archive_django_secret_key }}

ansible/roles/monero/README.md

@@ -62,10 +62,15 @@ Three things there are load-bearing:
 
 - **The whole device, no partition table.** One disk, one job; a partition table
   is just another layer that can disagree with itself.
-- **Everything after mkfs goes by UUID.** `/dev/sdX` ordering is decided by the
-  kernel at boot. This estate has been bitten eight times by an identifier
-  written down once whose thing then moved; `monero_data_device` is used *only*
-  to create the filesystem.
+- **The device name is read once, ever.** Each run first looks for a filesystem
+  labelled `monero-data`; if it exists, `monero_data_device` is not consulted
+  and the mount goes by that filesystem's UUID. Only a blank disk is found by
+  name, and that name is the Proxmox slot as published under `/dev/disk/by-id`
+  (`...drive-scsi1`), fixed before the VM boots and kept across `qm disk move`.
+  Never `/dev/sdX`: the kernel assigns those in probe order, and on 2026-10-04 a
+  reboot swapped `sda` and `sdb`, leaving the old `/dev/sdb` value naming the
+  root disk. The root-disk guard caught it - but a routine run should never
+  have been reading the name in the first place.
 - **`RequiresMountsFor=` in the unit.** `nofail` in fstab means a missing disk
   does not drop a headless VM into an emergency shell - but without
   `RequiresMountsFor` monerod would then start, find an empty directory, and

ansible/roles/monero/defaults/main.yml

@@ -38,16 +38,25 @@ monero_data_dir: /var/lib/monero
 # The whole device, no partition table: this disk has one job and a partition
 # table would only add a layer that can disagree with itself.
 #
-# monero_data_device is used ONLY to create the filesystem, once. Everything
-# afterwards - fstab, the mount, the systemd unit - goes by UUID, because
-# /dev/sdX ordering is decided by the kernel at boot and is not a stable name
-# for anything. This estate has been bitten eight times by an identifier that
-# was written down once and then moved.
-monero_data_device: /dev/sdb
+# The disk is known by two names, for two different moments:
+#
+# monero_data_label is its identity once it holds a filesystem. Every run looks
+# for it first, and if it is there the device name is never consulted at all -
+# a routine run cannot be broken by a device name, because it does not read one.
+#
+# monero_data_device is used ONLY on the first run, to find the blank disk and
+# create that filesystem. It must be a name fixed before the VM boots: Proxmox
+# passes the slot ("scsi1") in as the disk's serial, and udev publishes it under
+# /dev/disk/by-id. That survives reboots and `qm disk move` alike. NEVER
+# /dev/sdX: the kernel hands those out in probe order, and on 2026-10-04 a reboot
+# after moving this disk to the SSD swapped sda and sdb - the value written here
+# then named the ROOT disk.
+monero_data_label: monero-data
+monero_data_device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
 monero_data_fstype: ext4
 # -m 0: the 5% ext4 reserve exists to stop root-owned daemons deadlocking a full
 # root filesystem. On a dedicated data disk it is 25 GB thrown away.
-monero_data_mkfs_opts: "-m 0 -L monero-data"
+monero_data_mkfs_opts: "-m 0 -L {{ monero_data_label }}"
 
 # --- Network ----------------------------------------------------------------
 # P2P is OUTBOUND ONLY. No inbound port is published from the edge host, so the
@@ -95,15 +104,17 @@ healthcheck_push_url: ""
 # is set: Gatus rejects an unauthenticated push with 401.
 healthcheck_push_token: ""
 
-# WHEN TO CHECK. Daily, at a fixed hour, rather than an OnUnitActiveSec
-# interval that drifts with every restart.
+# WHEN TO CHECK. Every 5 minutes, on the calendar rather than OnUnitActiveSec so
+# the cadence does not drift with every restart. (It was daily while the node
+# did its initial sync on a rotating disk; since 2026-10-03 the chain is on NVMe
+# and fully synced, and get_info answers in well under a second.)
 #
 # Prefixed because the generic `healthcheck` role owns the unprefixed
 # healthcheck_on_calendar; an unprefixed default here would shadow it for any
 # play that includes both.
 #
-# Paired with the 26h Gatus heartbeat in
+# Paired with the standard 11m Gatus heartbeat in
 # infra/403_service_probe_registration.yml. Change one and you must change the
 # other: the window has to stay above the interval or the check alerts on its
 # own jitter.
-monero_healthcheck_on_calendar: "*-*-* 05:00:00"
+monero_healthcheck_on_calendar: "*:0/5"

ansible/roles/monero/tasks/storage.yml

@@ -5,10 +5,16 @@
 # appears nowhere in Ansible, so a rebuild of that host would come up with the
 # chain silently missing. This role does it in code instead.
 
-- name: Check the Monero data device
-  ansible.builtin.stat:
-    path: "{{ monero_data_device }}"
-  register: monero_device
+# Does the disk already carry our filesystem? Its label is the identity from
+# here on. When it is found, the device name below is never read: everything
+# that follows works from the labelled filesystem, so a stale or reordered
+# device name cannot break a routine run. blkid exits 2 for "no such label".
+- name: Look for an existing Monero filesystem by its label
+  ansible.builtin.command: "blkid -L {{ monero_data_label }}"
+  register: monero_labelled_fs
+  changed_when: false
+  failed_when: monero_labelled_fs.rc not in [0, 2]
+  check_mode: false
 
 - name: Find the device backing the root filesystem
   ansible.builtin.command: findmnt -no SOURCE /
@@ -16,48 +22,77 @@
   changed_when: false
   check_mode: false
 
-# A guard, not ceremony. mkfs on the wrong device is unrecoverable, and
-# /dev/sdX names are assigned by the kernel at boot - the whole reason
-# everything below this point works by UUID.
-- name: Assert the data device is a real block device and is not the root disk
-  ansible.builtin.assert:
-    that:
-      - monero_device.stat.exists
-      - monero_device.stat.isblk | default(false)
-      - not monero_root_source.stdout.startswith(monero_data_device)
-    quiet: true
-    fail_msg: >-
-      {{ monero_data_device }} is not a usable data device, or it is the disk
-      the root filesystem lives on ({{ monero_root_source.stdout }}).
-      Refusing to touch it. Check `lsblk` on the host.
+# First run only: the disk is blank and has to be found by name.
+- name: Prepare a blank data disk
+  when: monero_labelled_fs.rc != 0
+  block:
+    # The configured name is a /dev/disk/by-id symlink. Resolve it before
+    # comparing it to anything: the root source is a kernel name like
+    # /dev/sdb1, and a symlink path is never a prefix of one, so comparing the
+    # unresolved name would make the root-disk guard below always pass.
+    - name: Resolve the configured data device to its kernel name
+      ansible.builtin.command: "readlink -e {{ monero_data_device }}"
+      register: monero_device_path
+      changed_when: false
+      failed_when: false
+      check_mode: false
 
-- name: Read any existing filesystem on the data device
-  ansible.builtin.command: "blkid -o value -s TYPE {{ monero_data_device }}"
-  register: monero_existing_fs
-  changed_when: false
-  failed_when: false
-  check_mode: false
+    - name: Check the Monero data device
+      ansible.builtin.stat:
+        path: "{{ monero_device_path.stdout | default('', true) or monero_data_device }}"
+      register: monero_device
 
-# community.general.filesystem is idempotent and will not rebuild a filesystem
-# that is already the requested type, but it would happily create one on a disk
-# holding something it does not recognise. Say so out loud instead.
-- name: Refuse to reformat a disk that already carries a different filesystem
-  ansible.builtin.assert:
-    that:
-      - monero_existing_fs.stdout | trim in ['', monero_data_fstype]
-    quiet: true
-    fail_msg: >-
-      {{ monero_data_device }} already holds a '{{ monero_existing_fs.stdout | trim }}'
-      filesystem, not '{{ monero_data_fstype }}'. Refusing to overwrite it.
+    - name: Find the disk the root filesystem lives on
+      ansible.builtin.command: "lsblk -no PKNAME {{ monero_root_source.stdout }}"
+      register: monero_root_disk
+      changed_when: false
+      check_mode: false
 
-- name: Create the filesystem on the data device
-  community.general.filesystem:
-    fstype: "{{ monero_data_fstype }}"
-    dev: "{{ monero_data_device }}"
-    opts: "{{ monero_data_mkfs_opts }}"
+    # A guard, not ceremony. mkfs on the wrong device is unrecoverable.
+    - name: Assert the data device is a real block device and is not the root disk
+      ansible.builtin.assert:
+        that:
+          - monero_device_path.rc == 0
+          - monero_device.stat.exists
+          - monero_device.stat.isblk | default(false)
+          - monero_device_path.stdout != '/dev/' ~ (monero_root_disk.stdout | trim)
+          - not monero_root_source.stdout.startswith(monero_device_path.stdout)
+        quiet: true
+        fail_msg: >-
+          {{ monero_data_device }} (-> {{ monero_device_path.stdout | default('nothing', true) }})
+          is not a usable data device, or it is the disk the root filesystem
+          lives on ({{ monero_root_source.stdout }}). Refusing to touch it.
+          Check `ls -l /dev/disk/by-id` on the host.
 
+    - name: Read any existing filesystem on the data device
+      ansible.builtin.command: "blkid -o value -s TYPE {{ monero_device_path.stdout }}"
+      register: monero_existing_fs
+      changed_when: false
+      failed_when: false
+      check_mode: false
+
+    # No monero-data label was found, so a filesystem here is something else.
+    # community.general.filesystem would leave an ext4 of any label alone but
+    # happily format over a disk it does not recognise. Say so out loud instead.
+    - name: Refuse to reformat a disk that already carries a different filesystem
+      ansible.builtin.assert:
+        that:
+          - monero_existing_fs.stdout | trim in ['', monero_data_fstype]
+        quiet: true
+        fail_msg: >-
+          {{ monero_data_device }} already holds a '{{ monero_existing_fs.stdout | trim }}'
+          filesystem, not '{{ monero_data_fstype }}'. Refusing to overwrite it.
+
+    - name: Create the filesystem on the data device
+      community.general.filesystem:
+        fstype: "{{ monero_data_fstype }}"
+        dev: "{{ monero_device_path.stdout }}"
+        opts: "{{ monero_data_mkfs_opts }}"
+
+# By label: on the first run the filesystem was just created with it, on every
+# other run it was found by it.
 - name: Read the filesystem UUID
-  ansible.builtin.command: "blkid -o value -s UUID {{ monero_data_device }}"
+  ansible.builtin.command: "blkid -o value -s UUID -t LABEL={{ monero_data_label }}"
   register: monero_fs_uuid
   changed_when: false
   failed_when: false
@@ -67,18 +102,22 @@
 # without one. On a --check run against a disk that has no filesystem yet this
 # is expected and unavoidable: check mode did not create the filesystem, so
 # there is no UUID to read. Say that, rather than leaving a bare blkid rc=2.
-- name: Assert the UUID looks like a UUID
+# Exactly one: two disks with this label (a cloned disk, say) would make the
+# mount below ambiguous.
+- name: Assert exactly one UUID was found
   ansible.builtin.assert:
     that:
       - monero_fs_uuid.stdout | trim | length == 36
     quiet: true
     fail_msg: >-
-      No filesystem UUID on {{ monero_data_device }}.
-      {% if ansible_check_mode %}This is expected under --check when the disk
+      Expected exactly one filesystem labelled {{ monero_data_label }}, found:
+      '{{ monero_fs_uuid.stdout | trim }}'.
+      {% if ansible_check_mode %}No UUID is expected under --check when the disk
       has not been formatted yet - check mode cannot create the filesystem, so
       it cannot dry-run the mount either. Run without --check once, then
-      --check is meaningful from then on.{% else %}mkfs appears to have failed;
-      check `lsblk` and `blkid` on the host.{% endif %}
+      --check is meaningful from then on.{% else %}If none: mkfs appears to have
+      failed. If several: more than one disk carries the label. Check `lsblk -f`
+      on the host.{% endif %}
 
 # Same owner and mode as the "Own the mounted data directory" task below, and
 # that is deliberate rather than duplication. Before the disk is mounted this
@@ -93,7 +132,7 @@
     group: "{{ monero_group }}"
     mode: '0750'
 
-# By UUID, never by /dev/sdb. nofail so a missing disk does not drop a headless
+# By UUID, never by a device name. nofail so a missing disk does not drop a headless
 # VM into an emergency shell - monerod is stopped from starting by
 # RequiresMountsFor= in its unit instead, which the systemd monitor then reports.
 # noatime because LMDB touches a lot of pages and none of it wants an atime write.

ansible/roles/monero/templates/healthcheck.timer.j2

@@ -8,17 +8,13 @@ Description=Monero Node Health Check Timer
 
 [Timer]
 OnBootSec=2min
-# ONCE A DAY, not every 5 minutes.
-#
-# A 5-minute poll of monerod buys nothing a daily one does not: the node is a
-# multi-week initial sync on a rotating disk, and every failing poll is another
-# Signal message. The cadence and the Gatus heartbeat window are ONE decision -
-# the window MUST exceed this interval, and it is set to 26h in
-# infra/403_service_probe_registration.yml. A 24h push under a 24h window races
-# itself and alerts on jitter alone.
+# Every 5 minutes (see monero_healthcheck_on_calendar). The cadence and the
+# Gatus heartbeat window are ONE decision - the window MUST exceed this
+# interval, and it is the standard 11m in
+# infra/403_service_probe_registration.yml.
 OnCalendar={{ monero_healthcheck_on_calendar }}
-# Spread against the other daily pushes leaving this box.
-RandomizedDelaySec=900
+# Small: a delay comparable to the interval would eat the heartbeat's slack.
+RandomizedDelaySec=30
 Persistent=true
 
 [Install]

ansible/roles/signal_api/defaults/main.yml

@@ -4,12 +4,19 @@
 # Gatus does not speak Signal. It POSTs JSON to this service, which holds the
 # actual Signal identity and does the protocol work.
 
-# Pinned by digest for the same reason as Gatus: a tag is mutable.
-# Upstream publishes no versioned tags worth pinning to, so this pins the
-# DIGEST that `latest` resolved to when this was reviewed. `latest` is a moving
-# target; a digest is a content address, and `docker compose pull` either
-# fetches exactly this image or fails.
-signal_api_image_digest: "sha256:2399d449123cdad56c4d859277e3b9127e1a00c4d2ab4601c239882609286cf8"
+# Pinned by DIGEST, with the release tag alongside for humans - same as Gatus.
+# A tag is mutable; a digest is a content address, so `docker compose pull`
+# either fetches exactly this image or fails. Upstream publishes versioned
+# release tags (0.97 ... 0.101); the two values below must move together.
+#
+# THIS PIN EXPIRES. Signal's servers stop accepting signal-cli releases that
+# fall a few months behind. 0.100 (signal-cli 0.14.5, 2026-06-11) was refused
+# on 2026-10-18 - the phone reported "network error" and signal-cli logged
+# "Link request error: Connection closed!", with the network itself fine. A
+# stale pin fails exactly the same way for ALERTS: silently. Bump this at least
+# every couple of months: https://github.com/bbernhard/signal-cli-rest-api/releases
+signal_api_version: "0.101"   # bundles signal-cli 0.14.8
+signal_api_image_digest: "sha256:2cf09d66a86fdd210e42ee91662216364acc27da1163a8c44167a77ea3a6559a"
 signal_api_image: "bbernhard/signal-cli-rest-api@{{ signal_api_image_digest }}"
 
 signal_api_dir: /opt/signal-api

ansible/roles/xmrblocks/defaults/main.yml

@@ -74,8 +74,9 @@ healthcheck_push_url: ""
 # is set: Gatus rejects an unauthenticated push with 401.
 healthcheck_push_token: ""
 
-# WHEN TO CHECK. Daily, 30 minutes after monerod's own check so the two do not
-# contend for the same cold LMDB pages. See roles/monero/defaults for why the
-# name is prefixed and why this is paired with the 26h heartbeat in
-# infra/403_service_probe_registration.yml.
-xmrblocks_healthcheck_on_calendar: "*-*-* 05:30:00"
+# WHEN TO CHECK. Every 5 minutes, offset 2 minutes from monerod's own check so
+# the two do not land together. On NVMe a cold page render measured 29s and a
+# warm one 0.2s (2026-10-17); a 5-minute poll is what keeps it warm. See
+# roles/monero/defaults for why the name is prefixed and why this is paired
+# with the 11m heartbeat in infra/403_service_probe_registration.yml.
+xmrblocks_healthcheck_on_calendar: "*:2/5"

ansible/roles/xmrblocks/templates/healthcheck.timer.j2

@@ -7,14 +7,13 @@ Description=Onion Monero Blockchain Explorer Health Check Timer
 
 [Timer]
 OnBootSec=2min
-# ONCE A DAY, not every 5 minutes. See the note in roles/monero's timer: the
-# explorer follows the node it reads from, and this check costs a full page
-# render out of the LMDB (a cold one measured 68 seconds), so polling it hard
-# is not free either. The Gatus heartbeat window is 26h, set in
+# Every 5 minutes (see xmrblocks_healthcheck_on_calendar). The check renders a
+# full page out of the LMDB: 29s cold, 0.2s warm on NVMe - frequent polling is
+# what keeps it warm. The Gatus heartbeat window is the standard 11m, set in
 # infra/403_service_probe_registration.yml, and MUST stay above this interval.
 OnCalendar={{ xmrblocks_healthcheck_on_calendar }}
-# Spread against the other daily pushes leaving this box.
-RandomizedDelaySec=900
+# Small: a delay comparable to the interval would eat the heartbeat's slack.
+RandomizedDelaySec=30
 Persistent=true
 
 [Install]

ansible/services/arbret/deploy_arbret_edge_playbook.yml

@@ -0,0 +1,59 @@
+---
+# arbret's footprint on the edge host - and ONLY that.
+#
+# arbret itself (app, worker, Postgres, timers, its backup dump) is deployed by
+# the arbret owner's own Ansible repo since prd-arbret was merged into vipy on
+# 2026-10-17. This playbook owns the parts of the shared host that are ours:
+# the public front door, the tailnet port, and the backup directory contract.
+# See arbret_handoff.md at the repo root for the division of labour.
+- name: Publish arbret on the edge host
+  hosts: edge
+  become: yes
+  vars:
+    # Local port the arbret app binds on 127.0.0.1, per the arbret owner.
+    arbret_port: 3100
+    arbret_domain: arbret.com
+    # Admin port reachable from the tailnet only (ACL in headscale's acl.json.j2).
+    arbret_tailnet_port: 4117
+
+  tasks:
+    - name: Publish arbret through Caddy
+      ansible.builtin.include_role:
+        name: caddy_site
+      vars:
+        caddy_site_name: arbret
+        caddy_site_domain: "{{ arbret_domain }}"
+        caddy_site_upstream: "localhost:{{ arbret_port }}"
+
+    - name: Redirect www to the apex
+      ansible.builtin.include_role:
+        name: caddy_site
+      vars:
+        caddy_site_name: arbret-www
+        caddy_site_domain: "www.{{ arbret_domain }}"
+        caddy_site_body: |
+          redir https://{{ arbret_domain }}{uri} permanent
+
+    # tailscale0 only. UFW's default-deny applies to the tailnet interface too,
+    # so the headscale ACL alone does not open it - and this must never be
+    # reachable on the public interface.
+    - name: Allow the arbret admin port from the tailnet
+      community.general.ufw:
+        rule: allow
+        interface: tailscale0
+        direction: in
+        port: "{{ arbret_tailnet_port | string }}"
+        proto: tcp
+        comment: "arbret admin, tailnet only"
+
+    # The backup contract directory (arbret_handoff.md section 3.3). The arbret
+    # repo creates it identically; creating it here too means small-backups-box
+    # can pull - an empty dir - before arbret's first dump, so one missing
+    # source does not turn the whole pull job red.
+    - name: Ensure the arbret backup directory exists
+      ansible.builtin.file:
+        path: /opt/backups/arbret
+        state: directory
+        owner: root
+        group: backup-pull
+        mode: "0750"

ansible/services/forgejo/deploy_forgejo_playbook.yml

@@ -66,6 +66,10 @@
         owner: "{{ forgejo_user }}"
         group: "{{ forgejo_user }}"
         mode: '0644'
+        # Fresh install only: the web installer then writes the real config
+        # (INTERNAL_TOKEN, JWT secrets, DB path, ROOT_URL) into this file.
+        # Without force: false every re-run replaced it with this stub.
+        force: false
   
     - name: Download Forgejo systemd service file
       get_url:

ansible/services/gatus/deploy_gatus_playbook.yml

@@ -22,9 +22,10 @@
         number: "{{ signal_number }}"
         recipients: "{{ signal_recipients }}"
         default-alert:
-          # ── SIGNAL ALERTING IS OFF ──────────────────────────────────────
-          # Flip to true to turn the phone back on. Nothing else needs to
-          # change: this ONE key disables every Signal alert in the estate.
+          # ── SIGNAL ALERTING IS ON ───────────────────────────────────────
+          # Flip to false to silence the phone. Nothing else needs to
+          # change: this ONE key switches every Signal alert in the estate.
+          # (Re-enabled 2026-10-18 after relinking signal-api post-rebuild.)
           #
           # It works because `enabled` is the only alert field no registration
           # playbook sets. Gatus merges a provider's default-alert into each
@@ -36,7 +37,7 @@
           # Deliberately NOT done by deleting the `signal:` provider block:
           # that would strand ~90 endpoints referencing a provider that no
           # longer exists, and lose the number and recipients.
-          enabled: false
+          enabled: true
           # Overridden per group by the registration playbooks; these are the
           # values that apply if a caller sets nothing.
           failure-threshold: 1

ansible/services/headscale/deploy_headscale_playbook.yml

@@ -244,23 +244,9 @@
       vars:
         caddy_site_name: headscale
         caddy_site_domain: "{{ headscale_domain }}"
-        # Raw body, and it must stay raw: the /admin* route in front of
-        # Headplane is not expressible as a plain reverse_proxy. The previous
-        # version of this task wrote only `reverse_proxy localhost:8080`, which
-        # would have deleted the admin route and its auth on the next run.
-        caddy_site_body: |
-          @headplane {
-              path /admin*
-          }
-          handle @headplane {
-              basicauth {
-                  {{ headscale_ui_username }} {{ headscale_ui_password_hash }}
-              }
-              reverse_proxy http://localhost:{{ headplane_port }}
-          }
-          # Headscale API is protected by its own API key authentication
-          # All API operations require a valid Bearer token in the Authorization header
-          reverse_proxy * http://localhost:{{ headscale_port }}
+        # The Headscale API authenticates every request itself (Bearer API
+        # key), so a plain proxy is all it needs.
+        caddy_site_upstream: "localhost:{{ headscale_port }}"
 
   handlers:
     - name: Restart headscale

ansible/services/headscale/headscale_vars.yml

@@ -18,6 +18,3 @@ remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_h
 remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
 remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
 remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
-
-# Headplane (headscale admin UI), proxied at /admin* behind Caddy basic auth
-headplane_port: 3000

ansible/services/headscale/templates/acl.json.j2

@@ -26,7 +26,7 @@
       "dst": [
         "*:22",
         "tag:arbret-staging-box:4117",
-        "tag:prd-arbret:4117",
+        "tag:prd-vipy:4117",
         "tag:nodito:8006"
       ],
       "src": [
@@ -43,14 +43,10 @@
         "tag:mempool-box:8080",
         "tag:memos-box:5230",
         "tag:music-box:4533",
+        "tag:jar-lid-archive-staging-box:8000",
         "tag:monerod-box:18089",
         "tag:monerod-box:8081",
-        "tag:arbret-staging-box:80",
-        "tag:stalwart-box:25",
-        "tag:stalwart-box:465",
-        "tag:stalwart-box:993",
-        "tag:stalwart-box:4190",
-        "tag:stalwart-box:8080"
+        "tag:arbret-staging-box:80"
       ],
       "src": [
         "tag:prd-vipy"
@@ -79,10 +75,9 @@
     {
       "action": "accept",
       "dst": [
-        "tag:prd-arbret:22",
         "tag:prd-vipy:22",
         "tag:memos-box:22",
-        "tag:stalwart-box:22"
+        "tag:jar-lid-archive-staging-box:22"
       ],
       "src": [
         "tag:small-backups-box"
@@ -102,6 +97,9 @@
     "tag:fulcrum-box": [
       "{{ headscale_namespace }}@"
     ],
+    "tag:jar-lid-archive-staging-box": [
+      "{{ headscale_namespace }}@"
+    ],
     "tag:knots-box": [
       "{{ headscale_namespace }}@"
     ],
@@ -120,20 +118,11 @@
     "tag:nodito": [
       "{{ headscale_namespace }}@"
     ],
-    "tag:prd-arbret": [
-      "{{ headscale_namespace }}@"
-    ],
-    "tag:prd-monitoring": [
-      "{{ headscale_namespace }}@"
-    ],
     "tag:prd-vipy": [
       "{{ headscale_namespace }}@"
     ],
     "tag:small-backups-box": [
       "{{ headscale_namespace }}@"
-    ],
-    "tag:stalwart-box": [
-      "{{ headscale_namespace }}@"
     ]
   }
 }

ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml

@@ -0,0 +1,51 @@
+---
+# Jar Lid Archive on its own box, published from the edge host.
+#
+# See roles/jar_lid_archive/README.md. Per-host values (domain, port, seed users,
+# secrets) are in host_vars/<host>/, so a production host is a new inventory
+# entry in [jar_lid_archive], not a new playbook.
+- name: Deploy Jar Lid Archive
+  hosts: jar_lid_archive
+  become: yes
+  roles:
+    - jar_lid_archive
+    # "Does the app answer", as opposed to "are the containers up". /health/
+    # runs SELECT 1 and touches the bucket, and returns 503 if either fails.
+    - role: healthcheck
+      vars:
+        healthcheck_name: jar-lid-archive
+        healthcheck_description: "Jar Lid Archive /health/ (app, database and photo store)"
+        healthcheck_command: "curl -fsS --max-time 10 -o /dev/null http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
+        healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_{{ jar_lid_archive_instance }}/external"
+        healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
+
+- name: Publish Jar Lid Archive from the edge host
+  hosts: edge
+  become: yes
+  tasks:
+    # No basic auth: the one-off links (/links/...) are opened by people who have
+    # no account and no proxy credential. The app's own login guards the rest.
+    #
+    # Caddy sets X-Forwarded-Proto: https and overwrites any value the client
+    # sent, which is what lets the app trust it (DJANGO_BEHIND_TLS_PROXY).
+    #
+    # No request size limit is set, deliberately: Caddy has none by default, and
+    # phones upload multi-megabyte photos.
+    - name: Expose Jar Lid Archive over HTTPS
+      ansible.builtin.include_role:
+        name: caddy_site
+      vars:
+        caddy_site_name: "{{ hostvars[jla_host].jar_lid_archive_instance }}"
+        caddy_site_domain: "{{ hostvars[jla_host].jar_lid_archive_domain }}"
+        # Tailscale MagicDNS: the box has no public address.
+        caddy_site_upstream: "{{ hostvars[jla_host].ansible_host }}:{{ hostvars[jla_host].jar_lid_archive_port }}"
+        caddy_site_resolvers: "100.100.100.100"
+      loop: "{{ groups['jar_lid_archive'] }}"
+      # Not `item`: the role's own tasks loop, and would shadow it.
+      loop_control:
+        loop_var: jla_host
+
+    # The per-site validate cannot catch a conflict between sites.
+    - name: Validate the whole Caddyfile
+      ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
+      changed_when: false

ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml

@@ -0,0 +1,34 @@
+---
+# Jar Lid Archive backup: the database and the photo store, as ONE artefact,
+# dumped and age-encrypted on the box itself. small-backups-box pulls it.
+#
+# Only `web` is stopped. It is the only writer to both the database and the
+# photo store, so with it down both are quiescent and are captured at the same
+# instant - no row points at a photo the tar lacks. db stays up for pg_dump.
+#
+# storage is deliberately NOT stopped, though an earlier version did: on this
+# VM's contended HDD-backed disk SeaweedFS can take minutes to come back, and the
+# first test run left the app down for 26 minutes waiting on it.
+- name: Configure the Jar Lid Archive backup on its own host
+  hosts: jar_lid_archive
+  become: yes
+
+  tasks:
+    - name: Ensure Jar Lid Archive dumps itself, encrypted, on a timer
+      ansible.builtin.include_role:
+        name: backup_source
+      vars:
+        backup_source_name: "{{ jar_lid_archive_instance }}"
+        backup_source_description: "Jar Lid Archive ({{ jar_lid_archive_instance }})"
+        backup_source_dump_command: /usr/local/bin/jar-lid-archive-dump.sh
+        # Uncompressed: the bulk is JPEGs. See roles/jar_lid_archive/templates/dump.sh.j2.
+        backup_source_artifact_suffix: "tar.age"
+        backup_source_stop_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml stop web"
+        # `up -d --wait`, not `start`: it also repairs a db or storage that died
+        # meanwhile, and only returns once /health/ passes.
+        backup_source_start_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait --wait-timeout 300"
+        backup_source_retention_days: "{{ jar_lid_archive_backup_retention_days }}"
+        backup_source_on_calendar: "*-*-* 02:20:00"
+        # Reported to Gatus as backup-dump_<instance>.
+        backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_{{ jar_lid_archive_instance }}/external"
+        backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"

ansible/services/lnbits/deploy_lnbits_playbook.yml

@@ -85,6 +85,10 @@
         owner: "{{ ansible_user }}"
         group: "{{ ansible_user }}"
         mode: '0644'
+        # Seed a fresh install only. Without this every re-run replaced the real
+        # .env (AUTH_SECRET_KEY and all) with the example, logging every user
+        # out - and would have clobbered a restored one.
+        force: false
 
 
     - name: Configure LNBits environment variables

ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml

@@ -1,62 +0,0 @@
-- name: Deploy ntfy-emergency-app with Docker Compose and configure Caddy reverse proxy
-  hosts: edge
-  become: yes
-  vars_files:
-    - ./ntfy_emergency_app_vars.yml
-  vars:
-    ntfy_emergency_app_subdomain: "{{ subdomains.ntfy_emergency_app }}"
-    ntfy_emergency_app_domain: "{{ ntfy_emergency_app_subdomain }}.{{ root_domain }}"
-    # ⚠ UNRESOLVED: this app sends its notifications to an ntfy server, and the
-    # server it points at ran on watchtower, which is being destroyed. This was
-    # derived from subdomains.ntfy, which is now gone with it.
-    #
-    # Until an ntfy server exists again this URL is dead, and the app fails
-    # silently at exactly the moment it matters - it is an EMERGENCY notifier.
-    # Three ways out, none of them automatic:
-    #   * stand ntfy up somewhere else (the monitoring VPS, or vipy)
-    #   * point this at the public ntfy.sh
-    #   * retire the app
-    ntfy_emergency_app_ntfy_url: "{{ ntfy_service_url }}"
-    ntfy_emergency_app_ntfy_user: "{{ ntfy_username | default('') }}"
-    ntfy_emergency_app_ntfy_password: "{{ ntfy_password | default('') }}"
-
-  tasks:
-    - name: Create ntfy-emergency-app directory
-      file:
-        path: "{{ ntfy_emergency_app_dir }}"
-        state: directory
-        owner: "{{ ansible_user }}"
-        group: "{{ ansible_user }}"
-        mode: '0755'
-
-    - name: Create docker-compose.yml for ntfy-emergency-app
-      copy:
-        dest: "{{ ntfy_emergency_app_dir }}/docker-compose.yml"
-        content: |
-          version: "3"
-          services:
-            ntfy-emergency-app:
-              image: ghcr.io/pmartincalvo/ntfy-emergency-app:latest
-              container_name: ntfy-emergency-app
-              restart: unless-stopped
-              ports:
-                - "{{ ntfy_emergency_app_port }}:3000"
-              environment:
-                NTFY_TOPIC: "{{ ntfy_emergency_app_topic }}"
-                NTFY_URL: "{{ ntfy_emergency_app_ntfy_url }}"
-                NTFY_USER: "{{ ntfy_emergency_app_ntfy_user }}"
-                NTFY_PASSWORD: "{{ ntfy_emergency_app_ntfy_password }}"
-                UI_MESSAGE: "{{ ntfy_emergency_app_ui_message }}"
-
-    - name: Deploy ntfy-emergency-app container with docker compose
-      command: docker compose up -d
-      args:
-        chdir: "{{ ntfy_emergency_app_dir }}"
-
-    - name: Publish ntfy-emergency-app through Caddy
-      ansible.builtin.include_role:
-        name: caddy_site
-      vars:
-        caddy_site_name: ntfy-emergency-app
-        caddy_site_domain: "{{ ntfy_emergency_app_domain }}"
-        caddy_site_upstream: "localhost:{{ ntfy_emergency_app_port }}"

ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml

@@ -1,22 +0,0 @@
-# General
-ntfy_emergency_app_dir: /opt/ntfy-emergency-app
-ntfy_emergency_app_port: 3000
-
-# (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/)
-
-# ntfy configuration
-ntfy_emergency_app_topic: "emergencia"
-ntfy_emergency_app_ui_message: "Leave Pablo a message, he will respond as soon as possible"
-
-# Remote access
-remote_host_name: "{{ groups['edge'] | first }}"
-remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
-remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
-remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
-remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
-
-# Where the emergency notifications are sent. This pointed at ntfy.contrapeso.xyz
-# on watchtower; that host is being destroyed, so this MUST be repointed before
-# the app can work again. Left at the old value so the break is visible rather
-# than silently defaulted to something plausible.
-ntfy_service_url: "https://ntfy.contrapeso.xyz"

ansible/services/personal-blog/deploy_personal_blog_playbook.yml

@@ -45,6 +45,9 @@
         owner: "{{ ansible_user }}"
         group: www-data
         mode: '0664'
+        # Only on a fresh web root. Without this, every re-run (and site.yml)
+        # overwrote the real deployed index.html with the placeholder.
+        force: false
 
     - name: Publish the blog through Caddy
       ansible.builtin.include_role:

ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml

@@ -27,13 +27,16 @@
               container_name: vaultwarden
               restart: unless-stopped
               ports:
-                - "{{ vaultwarden_port }}:80"
+                # Loopback only. Docker writes its own iptables rules AHEAD of
+                # UFW, so a bare "8222:80" was reachable from the internet as
+                # plain HTTP, bypassing Caddy and TLS. Caddy is the only client.
+                - "127.0.0.1:{{ vaultwarden_port }}:80"
               volumes:
                 - ./data:/data
               environment:
                 WEBSOCKET_ENABLED: 'true'
                 DOMAIN: "https://{{ vaultwarden_domain }}"
-                SIGNUPS_ALLOWED: 'true'
+                SIGNUPS_ALLOWED: '{{ vaultwarden_signups_allowed | bool | lower }}'
                 LOG_FILE: /data/vaultwarden.log
 
     - name: Deploy vaultwarden container with docker compose

ansible/services/vaultwarden/vaultwarden_vars.yml

@@ -3,6 +3,13 @@ vaultwarden_dir: /opt/vaultwarden
 vaultwarden_data_dir: "{{ vaultwarden_dir }}/data"
 vaultwarden_port: 8222
 
+# Public registration. OFF: the accounts exist, and this vault is on the open
+# internet. It used to be hardcoded 'true' in the compose file and turned off by
+# a separate manual playbook - so every routine run of the deploy (site.yml
+# included) silently re-opened sign-ups. Flip to true only to add an account,
+# then back.
+vaultwarden_signups_allowed: false
+
 # (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/)
 
 # Remote access

ansible/site.yml

@@ -57,14 +57,15 @@
 - import_playbook: services/xmrblocks/deploy_xmrblocks_playbook.yml
 - import_playbook: services/memos/deploy_memos_playbook.yml
 - import_playbook: services/navidrome/deploy_navidrome_playbook.yml
+- import_playbook: services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml
 - import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml
 - import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml
 - import_playbook: services/headscale/deploy_headscale_playbook.yml
 - import_playbook: services/vaultwarden/deploy_vaultwarden_playbook.yml
 - import_playbook: services/forgejo/deploy_forgejo_playbook.yml
 - import_playbook: services/lnbits/deploy_lnbits_playbook.yml
-- import_playbook: services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml
 - import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml
+- import_playbook: services/arbret/deploy_arbret_edge_playbook.yml
 
 # ── Backups: each source dumps itself, the box pulls ────────────────────────
 - import_playbook: services/headscale/setup_backup_headscale.yml
@@ -72,6 +73,7 @@
 - import_playbook: services/forgejo/setup_backup_forgejo.yml
 - import_playbook: services/lnbits/setup_backup_lnbits.yml
 - import_playbook: services/memos/setup_backup_memos.yml
+- import_playbook: services/jar-lid-archive/setup_backup_jar_lid_archive.yml
 - import_playbook: playbooks/backups.yml
 
 # Deliberately not here. Every playbook in the repo is either imported above or

arbret_handoff.md

@@ -0,0 +1,456 @@
+# arbret on prd-vipy: monitoring and backups
+
+Handoff for the arbret owner. It covers what changed when prd-arbret was lost, and what
+your Ansible repo needs to do so that arbret is **monitored by Gatus** and **backed up to
+small-backups-box** from its new home.
+
+Written 2026-10-17, during the rebuild.
+
+---
+
+## 0. What changed, in one screen
+
+| | Before | Now |
+|---|---|---|
+| Host | `prd-arbret`, a droplet of its own | **`prd-vipy`**, shared with the personal_infra services |
+| Monitoring | Uptime Kuma (push URLs on `uptime.contrapeso.xyz`) | **Gatus** at `https://status.contrapeso.xyz`, running on `prd-spacey`. Uptime Kuma is gone for good. |
+| Backup destination | small-backups-box pulls `arbret@prd-arbret:/opt/arbret/backups/` | small-backups-box pulls **`backup-pull@prd-vipy:/opt/backups/arbret/`** |
+| TLS / front door | your own, on prd-arbret | **our Caddy on prd-vipy** (it owns 80/443) |
+
+Who does what:
+
+| Area | **You** (arbret repo) | **Us** (personal_infra repo) |
+|---|---|---|
+| App, workers, Postgres, analytics | deploy and run them | — |
+| Public HTTPS for `arbret.com` | tell us your local port | Caddy vhost, certificates |
+| Gatus endpoint definitions | — | all of them: Gatus config lives on prd-spacey, which you don't touch |
+| Pulled checks (HTTPS, `/api/health`) | keep `/api/health` honest | define and run them |
+| systemd unit checks (`arbret-server`, `arbret-worker`, `postgresql`) | keep the unit names stable | add them to vipy's monitored units |
+| Host liveness and disk | — | already covered for prd-vipy |
+| Analytics timer result | **push it** (§2.4) | register the endpoint |
+| Backup dump | **produce it to the contract** (§3) | pull it, retain it 90 days, check freshness |
+| Backup dump result | **push it** (§3.5) | register the endpoint |
+
+Your work comes down to three things:
+
+1. Live on a shared host without stepping on it (§1).
+2. Push two results to Gatus: analytics and backup.
+3. Write the dump to a new place, in a fixed shape.
+
+---
+
+## 1. Living on a shared host
+
+prd-vipy also runs Vaultwarden, Forgejo, LNbits, phoenixd and a reverse proxy for a
+dozen other services. Our Ansible converges it regularly. Anything below that your
+playbooks change will be reverted by ours, or will break ours.
+
+### Owned by personal_infra: do not manage these
+
+- **Ports 80 and 443 and the Caddy install** (`/etc/caddy/**`). Don't install nginx,
+  certbot or a second Caddy, and don't bind 80/443. Give us a port and we write the vhost
+  (see below).
+- **UFW and fail2ban.** The default policy is deny-incoming, with only SSH and 80/443
+  open. Don't disable UFW or reset its rules. If you need a port reachable from outside,
+  ask us.
+- **Users `counterweight` and `backup-pull`**, and their `authorized_keys`. In
+  particular **don't create, modify or authorise keys for `backup-pull`**: we create it
+  and it is shared by every backup on this host (§3).
+- **`/opt/backups/`** (the parent directory, `root:root 0755`). You own exactly
+  `/opt/backups/arbret/` and nothing beside it.
+- **Tailscale/Headscale membership** of the host, and the tailnet ACLs.
+- **Docker.** It's installed for our services. You don't need it (arbret is native
+  systemd); if you start using it, don't touch `/etc/docker/daemon.json`.
+- systemd unit names we use: `caddy`, `forgejo`, `lnbits`, `phoenixd`,
+  `*-backup.{service,timer}` for vaultwarden/lnbits/forgejo, `liveness-healthcheck`,
+  `disk-usage-healthcheck`, `service-health-healthcheck`. Prefix all of yours with
+  `arbret-` and there's no risk of collision.
+
+### Free for you
+
+- PostgreSQL. We run no Postgres on this host, so the system `postgresql` package and
+  cluster are yours.
+- `/opt/arbret/`, `/etc/arbret/`, the `arbret` system user, `arbret-*` units and timers.
+
+### Local ports already taken
+
+`22, 80, 443, 3000, 7657, 8222, 8333, 8765, 9740, 23334, 50002`. Bind your app to
+`127.0.0.1:<port>` on anything else and tell us the port. Our Caddy then serves
+`arbret.com` with a Let's Encrypt certificate and reverse-proxies to it, setting
+`X-Forwarded-For`, `X-Forwarded-Proto` and `X-Forwarded-Host`. Configure the app to
+trust proxy headers from `127.0.0.1` only.
+
+Port `4117` used to be reachable from one admin laptop over the tailnet. If that's still
+needed, say so and we'll add the ACL for prd-vipy. Don't open it in UFW yourself.
+
+### The host itself
+
+Debian 13, clock in **UTC**, tailnet name `prd-vipy`.
+
+---
+
+## 2. Monitoring with Gatus
+
+### 2.1 How it works here
+
+Gatus runs on prd-spacey and answers at `https://status.contrapeso.xyz`. Alerts go to
+Signal. There are two kinds of check:
+
+- **Pulled.** Gatus makes the request itself (HTTP, TCP, DNS) every N seconds and
+  evaluates conditions. Nothing is installed on your side. This is the right kind for
+  anything with a reachable URL.
+- **Pushed ("external endpoints").** Something on the host `POST`s a result to Gatus,
+  and Gatus alarms if a result is `false` **or if nothing arrives within the heartbeat
+  window**. This is the right kind for things with no URL: timers, jobs, internal state.
+  The heartbeat is what makes a dead timer visible: silence is a failure.
+
+Everything about an endpoint (name, group, token, heartbeat, alert rules) is defined in
+**our** repo, because Gatus's config files live on prd-spacey. Your side of a push check
+is just an HTTP call.
+
+### 2.2 The push API
+
+```
+POST https://status.contrapeso.xyz/api/v1/endpoints/{KEY}/external?success={true|false}&error={message}
+Authorization: Bearer {TOKEN}
+```
+
+- **`KEY`** is `sanitize(group) + "_" + sanitize(name)`, where sanitize lowercases and
+  turns any of `/ _ . , space # + &` into `-`. Use the exact keys in the table below.
+- **`success`**: `true` or `false`.
+- **`error`**: free text, URL-encoded. It's shown on the dashboard and in the Signal
+  alert, so on success it's fine to send something useful like `ok 31M`.
+- **`TOKEN`**: a bearer token we give you (§4). It's dedicated to arbret: it can only
+  write arbret's endpoints, and nobody else's token can write yours.
+
+Responses:
+
+| HTTP | Meaning |
+|---|---|
+| `200` | recorded |
+| `401` | wrong token for that key |
+| `404` | key not registered (yet). We register before you deploy. |
+
+Test it by hand once you have the token:
+
+```bash
+curl -i -X POST -H "Authorization: Bearer $TOKEN" \
+  "https://status.contrapeso.xyz/api/v1/endpoints/probe_arbret-analytics/external?success=true&error=manual%20test"
+```
+
+### 2.3 Your old Uptime Kuma monitors, mapped
+
+| Old monitor (prod) | New check | Gatus key | Kind | Who |
+|---|---|---|---|---|
+| `https`: front door returns 200 | `GET https://arbret.com/` → 200 | `public_arbret` | pulled, 5 min | us |
+| `health`: `/api/health` returns ok | `GET https://arbret.com/api/health` → 200 and `[BODY].status == ok` | `public_arbret-health` | pulled, 5 min | us |
+| `arbret-server`: unit active | systemd unit check on prd-vipy | `services_vipy-arbret-server` | push, every 5 min, by our host script | us |
+| `arbret-worker`: unit active | same | `services_vipy-arbret-worker` | same | us |
+| `postgresql`: unit active | same | `services_vipy-postgresql` | same | us |
+| `arbret-analytics`: timer ran OK | your analytics run reports its result | `probe_arbret-analytics` | **push, by you**, hourly; heartbeat 130m | **you** (§2.4) |
+| (new) beancount: daily job ran OK | your beancount run reports its result | `probe_arbret-beancount` | **push, by you**, daily 03:00; heartbeat 26h | **you** (§2.4, same hook) |
+| `arbret-backup`: timer ran OK | your backup run reports its result | `backup-dump_arbret` | **push, by you** | **you** (§3.5) |
+| prd-arbret liveness / disk | covered by prd-vipy's own host checks | `liveness_vipy`, `disk_vipy` | push | us |
+
+We also check:
+
+- `backup-store_arbret`: the backup **arrived** on small-backups-box, is fresh (<26h,
+  read from the filename timestamp), non-empty, and not suspiciously smaller than the
+  previous one.
+- `arbret.com` resolves to prd-vipy, checked every 6 hours.
+- `arbret.com` registration expiry, alarming 14 days ahead. This already existed.
+
+**Delete, don't port, every Uptime Kuma push script, timer and URL in your repo.** The
+endpoint no longer exists. A leftover timer will just fail silently every two minutes
+forever.
+
+The pulled health check relies on `/api/health` returning HTTP 200 with a JSON body that
+has `"status": "ok"` when healthy, and **a non-200 when it isn't**. If the body differs,
+tell us the exact shape. A health endpoint that checks the database is worth more than
+one that only proves the process answers.
+
+### 2.4 What you implement: the analytics push
+
+The cleanest hook is `ExecStopPost=` on the oneshot service the timer runs. systemd runs
+it after every run, success or failure, and sets `$SERVICE_RESULT` and `$EXIT_STATUS`
+for it. So no wrapper script is needed, and the run itself stays untouched.
+
+`/etc/arbret/gatus.env` (`root:root 0600`):
+
+```
+GATUS_PUSH_BASE=https://status.contrapeso.xyz/api/v1/endpoints
+GATUS_PUSH_TOKEN=<token from §4>
+```
+
+`/usr/local/bin/arbret-gatus-push` (`root:root 0755`):
+
+```bash
+#!/usr/bin/env bash
+# Usage: arbret-gatus-push <gatus-key>
+# Meant for ExecStopPost=, where systemd provides SERVICE_RESULT / EXIT_STATUS.
+# Can also be called by hand: SERVICE_RESULT=success arbret-gatus-push <key>
+set -uo pipefail
+KEY="$1"
+: "${GATUS_PUSH_BASE:?}" "${GATUS_PUSH_TOKEN:?}"
+
+if [ "${SERVICE_RESULT:-}" = "success" ]; then
+  success=true;  msg="ok"
+else
+  success=false; msg="${SERVICE_RESULT:-unknown} exit=${EXIT_STATUS:-?}"
+fi
+enc=$(printf '%s' "$msg" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g; s/=/%3D/g')
+
+# Never let a monitoring hiccup fail the job itself.
+curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \
+  -H "Authorization: Bearer ${GATUS_PUSH_TOKEN}" \
+  "${GATUS_PUSH_BASE}/${KEY}/external?success=${success}&error=${enc}" || true
+exit 0
+```
+
+In the analytics service unit:
+
+```ini
+[Service]
+Type=oneshot
+EnvironmentFile=/etc/arbret/gatus.env
+ExecStart=...your existing command...
+ExecStopPost=/usr/local/bin/arbret-gatus-push probe_arbret-analytics
+```
+
+Two things to get right:
+
+- `EnvironmentFile=` puts the token in the unit's environment, **not** in the unit file,
+  so the unit file can stay world-readable while the env file is `0600`.
+- **Tell us the timer's schedule.** We set the heartbeat to the interval plus slack: a
+  5-minute job gets an 11-minute window, a daily job gets 26 hours. If the schedule
+  changes later, tell us, or Gatus will either alarm on a healthy job or sleep through a
+  dead one.
+
+As Ansible, this is two templated files, an `EnvironmentFile=` line and an
+`ExecStopPost=` line. It's small enough to live in the role that deploys the analytics
+timer, rather than becoming a role of its own.
+
+### 2.5 Staging (arbret-staging-box)
+
+arbret-staging-box is a VM in our inventory, so its liveness and disk checks already
+exist. The same mapping applies with these keys:
+
+| Check | Key | Who |
+|---|---|---|
+| front door / health | `public_arbret-staging`, `public_arbret-staging-health` | us |
+| units | `services_arbret-staging-local-arbret-server` (and `-worker`, `-postgresql`) | us |
+| analytics | `probe_arbret-staging-analytics` | you, with the **staging** token |
+| backup dump | `backup-dump_arbret-staging` | you, with the staging token. Only if staging should be backed up; tell us. |
+
+The staging vhost (`arbretstaging.contrapeso.xyz`) lived on the old prd-vipy and was not
+in either repo. We're recreating it on our side.
+
+---
+
+## 3. Backups
+
+### 3.1 The pipeline
+
+```
+ 02:00 UTC  prd-vipy   your timer:  pg_dump | age -r <recipient>  ->  /opt/backups/arbret/arbret_<ts>.dump.age
+                       your script: POST backup-dump_arbret (success/failure)
+ 04:00 UTC  small-backups-box  rsync  backup-pull@prd-vipy:/opt/backups/arbret/  ->  ~/backups/arbret/  (keeps 90 days)
+ 05:30 UTC  small-backups-box  check-backups.sh  ->  POST backup-store_arbret  (then every 6 h)
+```
+
+- **The box only pulls.** Nothing on prd-vipy can reach into the backup box. The pull
+  logs in as `backup-pull`, an unprivileged account with a `restrict`ed key that can
+  read the backup directories and nothing else.
+- **The source encrypts.** Only the age **public** key is on prd-vipy, so the host can't
+  decrypt its own backups. The private key lives offline with the infra owner. Plaintext
+  never touches disk or the network.
+- **Retention is two-tier.** Keep about 7 days locally (the disk is shared). The box
+  keeps 90.
+
+Your existing dump role already produces the right **artefact**: a Postgres custom-format
+dump, age-encrypted, named `arbret_YYYYmmdd_HHMMSS.dump.age`. The last pre-incident one
+is `arbret_20261016_020001.dump.age`. Only its **location, ownership and reporting**
+change.
+
+### 3.2 What changes from prd-arbret
+
+| | prd-arbret | prd-vipy |
+|---|---|---|
+| Directory | `/opt/arbret/backups/` | **`/opt/backups/arbret/`** |
+| Account the box pulls as | `arbret` (your user, with our key authorised) | **`backup-pull`** (ours, already exists) |
+| SSH key for the pull | you authorised it | **nothing to do.** Remove that task, and don't authorise anything for `arbret` or `backup-pull`. |
+| Result reporting | Uptime Kuma push | Gatus push to `backup-dump_arbret` |
+
+### 3.3 The contract
+
+The checks on the box enforce these, and some failures are silent, so read every line.
+
+1. **Directory:** `/opt/backups/arbret/`, owner `root`, group `backup-pull`, mode `0750`.
+   Don't change `/opt/backups/` itself (`root:root 0755`, ours). The `backup-pull` group
+   exists before your playbook runs; if it doesn't, fail loudly rather than creating it.
+2. **Filename:** `arbret_YYYYmmdd_HHMMSS.dump.age`, timestamp in the host's local time
+   (UTC). The prefix must be exactly `arbret_`, because the box's prune and freshness
+   checks glob on `arbret_*`, and it reads the **dump time from the filename**.
+3. **Encryption:** `age -r <recipient>` in the pipe (§4). Nothing unencrypted is ever
+   written to disk.
+4. **Atomicity:** write `<file>.partial`, set ownership and mode, then `mv` it to the
+   final name. The pull skips `*.partial`. A half-written file under the final name
+   would be pulled and kept for 90 days as if it were a backup.
+5. **File ownership:** `root:backup-pull`, mode `0640`, on both the `.partial` and the
+   final file. If the pull account can't read a file, **rsync fails for the whole
+   directory**, so one bad file blocks every later night.
+6. **`set -euo pipefail`.** Without `pipefail`, a failing `pg_dump` piped into a
+   succeeding `age` exits 0 and produces a tiny, valid-looking, useless artefact.
+7. **Timing:** finished before 04:00 UTC. 02:00 is good.
+8. **Local prune:** delete `arbret_*.dump.age` older than ~7 days, plus stale
+   `arbret_*.partial` files at the start of each run. Put nothing else in the directory:
+   no subdirectories, logs or checksums.
+9. **Report** the outcome to Gatus on every exit path (§3.5).
+10. **No service stop is needed.** `pg_dump` takes a consistent snapshot online. Don't
+    stop `arbret-server` for it.
+
+### 3.4 Reference script
+
+This is the same shape as the script every other service on the host uses (our
+`backup_source` role), adapted to `pg_dump`. Adjust the database name and the
+`pg_dump` invocation to match your current role.
+
+`/usr/local/bin/arbret-backup.sh` (`root:root 0750`):
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+umask 077
+
+BACKUP_DIR=/opt/backups/arbret
+RECIPIENT="age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza"
+RETENTION_DAYS=7
+NAME=arbret
+DB=arbret                                   # <- your database name
+ARTIFACT="${BACKUP_DIR}/${NAME}_$(date +%Y%m%d_%H%M%S).dump.age"
+
+PUSH_URL="${GATUS_PUSH_BASE:-}/backup-dump_arbret/external"
+PUSH_TOKEN="${GATUS_PUSH_TOKEN:-}"
+
+report() {   # $1=true|false  $2=message
+  [ -n "${GATUS_PUSH_BASE:-}" ] || return 0
+  local enc; enc=$(printf '%s' "$2" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g')
+  curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \
+    -H "Authorization: Bearer ${PUSH_TOKEN}" "${PUSH_URL}?success=$1&error=${enc}" || true
+}
+on_exit() {   # the ONE exit trap: bash keeps only one, a second replaces it
+  local rc=$?
+  if   [ "$rc" -ne 0 ];       then report false "arbret dump exited ${rc}"
+  elif [ ! -s "$ARTIFACT" ];  then report false "arbret produced no artefact"
+  else                             report true  "arbret $(du -h "$ARTIFACT" | cut -f1)"; fi
+}
+trap on_exit EXIT
+
+command -v age >/dev/null || { echo "age not installed" >&2; exit 1; }
+getent group backup-pull >/dev/null || { echo "group backup-pull missing - personal_infra has not provisioned this host" >&2; exit 1; }
+
+install -d -o root -g backup-pull -m 0750 "$BACKUP_DIR"
+rm -f "${BACKUP_DIR}/${NAME}_"*.partial
+
+runuser -u postgres -- pg_dump -Fc "$DB" | age -r "$RECIPIENT" -o "${ARTIFACT}.partial"
+chown root:backup-pull "${ARTIFACT}.partial"; chmod 0640 "${ARTIFACT}.partial"
+mv "${ARTIFACT}.partial" "$ARTIFACT"
+
+find "$BACKUP_DIR" -maxdepth 1 -type f -name "${NAME}_*.dump.age" -mtime +"$RETENTION_DAYS" -delete
+echo "wrote $ARTIFACT"
+```
+
+Units:
+
+```ini
+# /etc/systemd/system/arbret-backup.service   (0644; the token is in the env file, not here)
+[Unit]
+Description=arbret database backup
+After=postgresql.service
+
+[Service]
+Type=oneshot
+EnvironmentFile=/etc/arbret/gatus.env
+ExecStart=/usr/local/bin/arbret-backup.sh
+
+# /etc/systemd/system/arbret-backup.timer
+[Unit]
+Description=arbret database backup, nightly
+
+[Timer]
+OnCalendar=*-*-* 02:00:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target
+```
+
+`age` and `curl` come from Debian (`apt install age curl`). Installing them from your
+playbook is fine, since both repos asking for the same package doesn't conflict.
+
+### 3.5 Reporting the backup
+
+The script above already reports, on every exit path, to `backup-dump_arbret`, using the
+same `/etc/arbret/gatus.env` as the analytics push. Its heartbeat is 26 hours: one
+missed night alarms. The box-side check (`backup-store_arbret`) independently notices
+within 6 hours of the next pull if the artefact stops arriving or stops being fresh.
+So a broken dump and a broken pull show up as **different** red lights.
+
+### 3.6 Verify after your first deploy
+
+```bash
+sudo systemctl start arbret-backup.service
+sudo journalctl -u arbret-backup.service -n 20 --no-pager
+ls -l /opt/backups/arbret/                     # root backup-pull, -rw-r-----
+sudo -u backup-pull head -c 30 /opt/backups/arbret/arbret_*.dump.age   # must print "age-encryption.org/v1"
+systemctl list-timers arbret-backup.timer
+```
+
+Then check that `backup-dump_arbret` is green on the status page, and tell us. We'll
+trigger a pull rather than wait for 04:00, and confirm `backup-store_arbret`.
+
+### 3.7 Restoring
+
+Only the infra owner can decrypt: the age identity is offline. They'll hand you a
+decrypted dump over a secure channel. The latest pre-incident one is
+`arbret_20261016_020001.dump.age` (31.9 MB, sha256 `2bb771957513654cb907e5d63153e7714d54c7d5695d7caf09a05f948dcf515d`).
+Anything written to arbret after 2026-10-16 02:00 UTC is not in any backup.
+
+For a custom-format dump:
+
+```bash
+sudo -u postgres createdb -O arbret arbret          # or let --create do it
+sudo -u postgres pg_restore -d arbret --no-owner --role=arbret -j 4 arbret_20261016_020001.dump
+```
+
+Create the roles the app expects first. `pg_dump -Fc` doesn't include them. Then shred
+the plaintext dump: `shred -u arbret_20261016_020001.dump`.
+
+---
+
+## 4. What you'll receive from us
+
+| Item | Secret | Used for |
+|---|---|---|
+| Gatus push token, **prod** | yes | `/etc/arbret/gatus.env` on prd-vipy. Keep it in your vault. |
+| Gatus push token, **staging** | yes | the same file on arbret-staging-box, if you do the staging pushes |
+| age recipient `age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza` | **no**, it's a public key | the backup script. Safe to commit in plain text. |
+| SSH access to prd-vipy for your playbooks | access | being arranged separately |
+| The decrypted 2026-10-16 dump | yes, it's the full prod DB | restore, handed over once |
+
+You need **no** SSH key for the backup pull, and nothing from the old Uptime Kuma setup.
+
+## 5. What we need from you
+
+1. The local port arbret listens on (for the `arbret.com` vhost), and whether
+   `www.arbret.com` should be served or redirected.
+2. The exact `/api/health` response when healthy, and confirmation that it returns a
+   non-200 when the DB is down.
+3. The analytics timer's schedule (sets the heartbeat).
+4. Whether port 4117 over the tailnet is still needed.
+5. The final unit names, if not `arbret-server`, `arbret-worker`, `postgresql`.
+6. Whether staging should be backed up, and whether staging pushes are in scope.
+7. Who besides the infra owner should get Signal alerts for arbret.
+
+Once we have 1 to 3, we register the endpoints, the vhost and the pull source. Then you
+can deploy and see green on the first run.

tofu/nodito/vms.auto.tfvars

@@ -90,6 +90,14 @@ vms = {
     disk_size_gb = 20
     ipconfig0    = "ip=dhcp"
   },
+  "jar-lid-archive-staging-box" = {
+    vmid         = 3300
+    name         = "jar-lid-archive-staging-box"
+    cores        = 1
+    memory_mb    = 1024
+    disk_size_gb = 10
+    ipconfig0    = "ip=dhcp"
+  },
   # disk_size_gb here is the music library's ceiling, not just the OS: the
   # library lives on the root disk at /srv/music. ~45 GB is usable after
   # Debian, so roughly 300-450 albums in FLAC. Growing it means a Proxmox