From c3a7ca0ba785daa9e8aa1d588e5f156eefa4c69c Mon Sep 17 00:00:00 2001 From: counterweight Date: Sat, 3 Oct 2026 23:56:30 +0200 Subject: [PATCH 01/15] jar-lid-archive: a staging deployment on its own box MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/group_vars/all/main.yml | 5 + ansible/group_vars/all/vault.yml | 430 +++++++++--------- .../host_vars/jar_lid_staging_local/main.yml | 28 ++ .../host_vars/jar_lid_staging_local/vault.yml | 34 ++ ansible/infra/402_public_monitoring.yml | 5 + .../infra/403_service_probe_registration.yml | 1 + ansible/inventory.ini | 8 + ansible/playbooks/backups.yml | 7 +- ansible/roles/jar_lid_archive/README.md | 121 +++++ .../roles/jar_lid_archive/defaults/main.yml | 57 +++ .../roles/jar_lid_archive/tasks/deploy.yml | 100 ++++ .../roles/jar_lid_archive/tasks/docker.yml | 74 +++ ansible/roles/jar_lid_archive/tasks/main.yml | 6 + ansible/roles/jar_lid_archive/tasks/seed.yml | 13 + .../templates/docker-compose.yml.j2 | 123 +++++ .../jar_lid_archive/templates/dump.sh.j2 | 25 + .../roles/jar_lid_archive/templates/env.j2 | 5 + .../services/headscale/templates/acl.json.j2 | 7 +- .../deploy_jar_lid_archive_playbook.yml | 51 +++ .../setup_backup_jar_lid_archive.yml | 34 ++ ansible/site.yml | 2 + tofu/nodito/vms.auto.tfvars | 8 + 22 files changed, 929 insertions(+), 215 deletions(-) create mode 100644 ansible/host_vars/jar_lid_staging_local/main.yml create mode 100644 ansible/host_vars/jar_lid_staging_local/vault.yml create mode 100644 ansible/roles/jar_lid_archive/README.md create mode 100644 ansible/roles/jar_lid_archive/defaults/main.yml create mode 100644 ansible/roles/jar_lid_archive/tasks/deploy.yml create mode 100644 ansible/roles/jar_lid_archive/tasks/docker.yml create mode 100644 ansible/roles/jar_lid_archive/tasks/main.yml create mode 100644 ansible/roles/jar_lid_archive/tasks/seed.yml create mode 100644 ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2 create mode 100644 ansible/roles/jar_lid_archive/templates/dump.sh.j2 create mode 100644 ansible/roles/jar_lid_archive/templates/env.j2 create mode 100644 ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml create mode 100644 ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml diff --git a/ansible/group_vars/all/main.yml b/ansible/group_vars/all/main.yml index 979d54c..3a365f8 100644 --- a/ansible/group_vars/all/main.yml +++ b/ansible/group_vars/all/main.yml @@ -62,6 +62,11 @@ subdomains: # has nowhere to carry a second credential. Same call made for monero above. music: music + # Jar Lid Archive staging (jar-lid-archive-staging-box, proxied via vipy). + # Public with no basic auth: the one-off photo links are for people without + # an account, and the app has its own login for everything else. + jar_lid_archive_staging: jars-staging + # Read by plays across several groups, so global rather than group_vars/. headscale_namespace: counter-net diff --git a/ansible/group_vars/all/vault.yml b/ansible/group_vars/all/vault.yml index c5dc402..dd06bdc 100644 --- a/ansible/group_vars/all/vault.yml +++ b/ansible/group_vars/all/vault.yml @@ -1,214 +1,218 @@ $ANSIBLE_VAULT;1.1;AES256 -66356262646437623239383039643762363139653565356263623438303833663431396566353163 -3631323563343638346464386661346235656130383238370a353236623366313561646334363639 -35346232376138666430363437346638326264646562386436323962633630663136626462343762 -3862333139646636650a656337306533313436333239653561333231363030633339303835396637 -61633763313464353933306333383061346563623637383166646332303237653065346233623134 -39386130613866386235376266643062653833643132633235393931636366376164306331303762 -33393562303738393364323465653163633730633636343764646337383733643362396132303063 -32336461396561663063633137613234356535383863346334666465643437613962376439356433 -66353532333933343462636163353637613637356530333331636363616265376263636538656566 -62643137623933363237343865613131383766633561636664616434393538646332346132336630 -62333361626339626638333666393735373834396431343965386532313834353837336664306530 -36303138373734343735653637663466353435306166316661353738353035366535333265626330 -36396538353566316634376662383564323861303063386163663061323435656261316439336539 -65323063343961646435373134303039373139333265633838313636373463316231633832306261 -36623934343731343938306332356261646264613063393834336332666230626534653464363335 -66356331666434623735613333303466656263313964363030316231303330303735373135653763 -30323635303435656639653037366430316330333033306339366361353765656337316561366263 -38613265363565373832636239396362616435303536646465326132626239353361613137626632 -30353764323430656631366134373263343861376666346166363131333137376466393438346461 -31613435363637306432343437653636356665373838626361306332316664343266356666353238 -35303464326633326438353730393462373437313864326638323262323034346464643038386462 -34613435326432663566386331333430663336656239353131633261626637613035636662346136 -33626131626435316566393831333534393663326465633531373932626631613063666165613766 -63636463656435386333323037656533306331646561363061613537663738326162656332393161 -62313338376630643031666630663337646439623862306663303335303232306161353336643334 -30386537336265383334336431653637303836376261303038653739396263666161653365356133 -62636638393465363239616264316331316139363063313431323165313237646437643361653539 -63393932623934313062376137626266363330623766646231353561646332353565643638356633 -39656565363962653038393430363666643361343131323231663566343035643631646433306464 -64343735393937333036623139303531343338373162396635613863316466393235343865333065 -63353662313737353737353736316261346466366666666536653064356130326334326137383933 -64306331663430313838376266613763343537613965303937393364396238663266363539313531 -33616133646365396336613234636630396239316633333832653138393964616261643531663836 -36666265386134383430626637643766636439393636336436366265396239303335306563653962 -62323763626265373037343766386433646637633838643933633738333037633336646663316335 -36373534663439306137353236343565636538636333313239373936653263366637613335663233 -32306536643866656461343666396565373661353561393665643162613539633634666138626266 -35613663376465626363373864626266393335396461316338653262316131643633666361333537 -64366533633535313633306535393732383436333762616539646463366665623866323463336263 -38313365326338386664363538633232343636303135356230393065613238386366323435633931 -37353537313330396463383233616532653139376263346264623366383339306139386231636466 -35383761373733386133386436316431613265306264656630353561333736613734653164333031 -32643237366534643062653065333632343563633034663266393961353763663534353466383561 -38373332323135353932343231353433366637333437636238623235663836393735346230623438 -37616639643231396564373033636162616234366262333461343139363539316332333431636565 -64396639383462386239613035303565376163633637653137373631383864353532373465626630 -63636531653063306131623735393963623836313566643565386336636565633130383264316533 -33343030316365373164663661306161363062646361646332393935373533626331623163323038 -61316666366263313531373530366639333364336563623162323938356662626565383363303533 -61323730313161633035646435346234366661386339346663363438663862636361343033306134 -66396365653634373235623837666537323230616334663263633135316433313734663863333561 -34326134613335346661316632353764343235326630316535306666326662613436383764646366 -33616135333038333834656439383733656436663433303536313961333662366564376338326661 -35393839643663333130383965663538393931363662336165636435643132353461343266363265 -66346635313337346661646461386336386461346133323435643939616234666665616261313731 -38383265643830646364373164656133396132666236313265346337643939613831303366393530 -63333330346131656134366237323661316534323330306330616564393130353136306461666361 -62376637646230623264363437313161396130616231346133346665303834623833663062663436 -30643736376638303065356161366561353565343139343731313263393933313239363363636638 -32663932626332626366333635616335316235363561373539623265663733373030373439373831 -65333763653635613964396437333833626535643666643433393031336461366562333934666234 -62383632303436633965643430363763353361613762306361613066356136383131383232313961 -30626531353933336563656139316338616565303261363933306164393966343037353431653233 -32636631373966323838333666313964323933306638653162646166623237343662646134383337 -37383039383531656337626334386638613133343133633637343738343731326663356666333161 -61323765636135306563323064346465633537613361383761393361353363396237333438306166 -35386338356135633365396661356535623232613765653931303063353538643130323439366438 -39363936316263333562646133646464346362343761336662353764613261396364626164313664 -31383839303361323461663662393736616363646232666633323134653138306161323735386638 -30333365663064366535656134323338663566616163333634353939363533356364393563363333 -65623838323034336530326363383237373563326334343066323738663833353930663930373965 -30363766633137396561376636636566363862353062663038653534623430643766633631356562 -63623762343433626363353239376539633136333038623338303961373938396632313966666532 -33633865626462636137343436386661353561623035656435623566613530623163316465383664 -37346261616239653138623262623930316432633531313037353938396164646534316564613238 -36643137303965316238346134323639386261633338363130323134336632636564636638653861 -38346566643966386465366535393166626463313039613236393831393732613537336530393936 -31613234333836333732323165653931336565316564346239336663363736323763623962383261 -65626332626565636162336264303964323139303831656262373161303861393366353237393136 -62303361636131646335633733373966626135396630623133623733373565306234646165656138 -35383065613365363438396332306236646432663731326666613838643434393866623163316631 -33633931373135336438643735646266383462386431666330363066386538386335383231653334 -64323366363032386662333662613562333864356238626161366566356564343339343838373437 -30363133363136663036656461343534666636343932313336336637376665393564313234373430 -39356362616539366237646363646364326664653232313235653831663539353530393261346532 -38653134623732313466386635313730303639613561323665356633653836393237333737323864 -36323736326239636565346536646566656138666630396366353930383237333338626532663131 -64366339303230663530626632623066613030663533366539313866646466373439663831363936 -36366636653032343064383661383664636463343165326532653861663537633238323937363839 -65333030653566306465643035633162393534323731346330333861353236393732346130313034 -64303736323030643533633539623638643232306435663238393566376564313162633334613462 -33363931383664633433363032643536356135626662653733303339656339373735643732626535 -35316131356135393933393462336163373062653133356133393830393638313838396361336635 -62323463333464353939346233333462646161633036643332313337353065376561336630383163 -38306666666634393766366436653436623138363830343636323833353365333762343533633835 -66363730383932303661313831663566613830393530616164323963636339393339623065316534 -38373662636330336632653335316463623935653961363566396132616433633831636334306236 -34393761663962623563333534383739613965353033646235653532303330343961656639333337 -31643462663538313930383038343964326463363935396365343466643236653738626639616434 -62343337356239316162383564666131663535313633373062353030336439396533376235386131 -31653865643931323364356566316435333336316166643063613164643266313834323235353863 -34623936613366303161353165386662376465656366656561613466383830326162336238656236 -30646632396231653263373631346330376461303231373733646231373462623131303633616166 -37366435353830303234393962663334656335356661333437363637663664323430623135346262 -33303233343362336261633663336635366233313466636233333836663464633637383964313133 -31363461336237373337353438396132616134303432336533653737333538383534363735316334 -62303666363532383131383833383536333733306163366339653038313930646531653565386335 -38313561303062336232333634343866393763636131333665396430306432373734613834316636 -61306138626530616230343462613036323166656334656637616139663030623932386132393762 -36363364343737613165376365323361363233616532643866303632653063343431613338373633 -63303263626434326539316439323239616632653632346538336535373336623739323931623462 -35656365336630323035616161656237353135373366376466326264323863336166633438663834 -63646562386339363235396163646334333238323932383230643964366466323064313138323464 -62636239663635336332633334656631383261306437333963363464303430666462633333666638 -37633830663439396133323861303735346561376434386131363864316332623137376139376231 -30303765646132306261363335366565306564373261663336613336336632616231363366316135 -65303966616439643863326662656439623938373833323738646535636230663136653266653533 -63326235376633333366373436333534336436366434363461393763633437616262326133623563 -37613161303161356632316335616639373564306161623339353066383163303431313333306635 -36326435376661343237303633316462613532313232643461313065376134393338613130336161 -38326463303562666662613361656161306139653431386630303561626666613037393238313463 -62623533396532326332326636346134616236653638363837643131343166333564323538653638 -63383261383265353331663864643933373562396565346163396434383765343962336363353261 -38353034643436626632623235396261326235323937646135326532313561396361363062333366 -32343666343163636233646466613633383034653438646262666133643830333834383233343136 -33646437396137613466643031323538626332613066333065643237353263636435393964356464 -34383265303364663233326432313230336436373866303435336666343832383737303030306534 -36383036363937393965386431343561333961663565333565333261323033616630343130613934 -32383166333063623235333464343864326531333165346261643131393264346534393035393431 -64636139383232383332623866323363623666326361386134353030376366323039393033653061 -63383338303734323637386366646365663032636463336261643533663638343661613462373736 -32663566666133656165396233643334343335333731333938376236633330366262366139326133 -32653163343963333739303061643936363861313938646164316564393261663765393462613837 -31646636326330663464363034376230393739343961653130303166366233396262656634313061 -65646664373161356134616165383265346633376136376132643234623334353563323539333065 -61623265343730326164373063383937323766363466343466316364373935626138343938323261 -38323739663131346138623663393832346236356631633133313436363364663536316561623631 -32393562646666343537663039363165646337653564363065353433643031353865386361613534 -37633639303636313261613563353136346465326163623361323230306364396362373966336262 -31353565623239343564333533656236656666316638646666376138373164663138633835366162 -37633036346562663261343731366161613361656534343839303062373232636339643961666432 -61396665656638333435386338383638323664313139656331336131326563373235646539633835 -30393235316564656131303932386336366661373737393264613364663030363862656462646463 -33303766303166336132626265653431313263363131653835336634356632666230666235383262 -31653837356536666266383562633437376161653664353533656331303563626539626161653562 -64333337626430616330346233613566353330383361353234376236656336653735663838343333 -37643162376630373531396138643861326562323133303363363837306138373434306364356364 -64623037643737663138626262326630323638376336663062393930663566316632323862653135 -66363664336431373966663133353030393863313538366632643932363438613035326563323163 -63343565373862316230643166336438393464323032666334653631393761303337303861666134 -36663637346136623564613233343432383930616339626161623933323565623861653939326165 -37383436353533613736323237666433316130613464396333343965393031333138396461376632 -61373166643766376234646436653566313637326434363261333831336136306137333061323035 -66303335363035666366386364303861383231366339643666313636623733616562333539643465 -32353566326262326637393434656133313963646435333135613630313865373430366265666632 -37316539363439663131323835313761313765653564663337346462653332613430373037333064 -64323731623566326664613634626265383661346462323463303638346464643366663333336339 -65626265663230346338633764633436653339373031623438633638326137613635633239343834 -35373132376133363861373062386235353836633030323236343666323134326135336231613566 -34326263643163633935643065306165643564663831633561353061653331326566633664376432 -33353835366265346266663836326465356166626432356663366566313962663862363930336334 -65323337386231376361343933353865633330613466323337306137633762653931313331313339 -36663965353832386662666433383934303737656336383432633632646135383835336631373861 -63646164373137316662373330333465343134633466396566333334663036663039373934333832 -35343661303565646464373834646234623837363562386463383237373533316638666663613633 -30313437626231343062636437363836386635333862373362643532646335353863323936663034 -30316461373330623130353563643035303561303666393464636533626562633165393236663936 -31313061303465633332646363643866306439383636356366623565663437313365376331656666 -34333930333235306636646664326561613165633233353937666238646536323131313232626366 -36623237613635343134313366346661656433643136353433393532303165326564646331333035 -35353365633039353161663337306463303535643062366232336239656162653532306539333035 -65373735373832326337333962643330363338656134303637313337393466366263363264343631 -30656433643562393537326361323733643639396537303065623936336230346664616534323262 -36313864396364353836373538333130303731383437376137383938383633373063323936613464 -37613166323432363232313834326565663265336138396335313230303935396266646438323833 -65306666636338373239346332613464373235613137643133626261313031326462356138356632 -66363465346165306533613636383938636531656631306139666237396632646634333937623032 -33613439656637383537356364336265653931613835333734383836343930313838626564333065 -35363735313162616238643438316535356663396339316331633232393232316131396432653639 -36303335643534376536383331323562353139613130383330636334636261326462656565643138 -63326365363163383932653562363833643761653839306131343737353633313737353732633463 -62326563336339383661376665353230613366646132653364396338663337326136333263363838 -35626537666238383435613431666365303133373166386136353061373765313632343665306439 -34356139363636623639626633623030363933303539626665383733363638303832333561653130 -64393437386362333731643131663263626434666661326533386362306362653732376366626139 -65626662323232636134653334366639386164653535323435613863353332366630303636663339 -34333633373737333732363664353437613762383064316134393139393539306564396237343332 -64636339643030353365356136366264353935643830633237316532316265316635393664326533 -64343030396334326135336465336637623832363065353163316465303537643861306632336535 -31366338323263313965643565656430366566393130326333383862313239356437623232613865 -63383863316639346632323834633739356464313865376463323633393365653033376431373138 -61363035653233373036666531303331383337353839363563386333313730363664666463333735 -61383638363164336265383664376463616536383533666265383961353961303833353265366633 -32646330333336613631633830633837346332653437356536363532396438326364623465383133 -36656231343734643830383335623930343132386336666366623161666637623862363033373236 -32666332316266623230666231663162643262653231313965323137383532633663323239396139 -37656533373065623161623766333335353965373731346639613566343236363239313537636239 -34626533346138613764346161383337623265353563343066303236333534316137356636396565 -32616136306665346566616565346162613736333836623363346232373932366365663531653435 -37623539366233366539396361386163643161653731353232313034613135373039653339346637 -31623138326133366333656661393837653939343064656638616630346462363065623139323131 -65363036383433663566626336353432613535363239373264356237323032323634383662613763 -39646161363639326330346332353965366330346632303939623136303238323431346361656462 -61623838626139636163353430636461666239343337373132643334616234616339313761353331 -36643165373834373437363165303861306135383236303030626463373235393234313765343563 -37353932366430666539623230373161303935633565323366363232336264643961383835333066 -32623366663965316662383036306462616465353139336333643936363838313538363832363330 -33313861313636373831656431383434363664663734626363646365666135376331366335653064 -30386235313037363365363232316338613634643235383037393635333166346632356435373039 -66663862373939346434333130383065386135306661353066343330666563386136376562316134 -63346435643132316434363666343964303335663131383831373430653632663638396333393932 -36383664376666393761643730306336333538376637633135616166346432636464336434323031 -3966 +33653731643464333331646636613630383836323134363761333038383334353230663631376666 +3861376232363238613234663263646137336662666263360a633661313435616338383337396664 +31316234616365326363353739323230326264623562313061303339373265663435313938326634 +3464636635386664620a663330396463653963613537653230333430326137333964386530343636 +62383336343031363537383362643934643839653731346364636534373430303636353431646534 +66313264343338623333633964376137623833613664393137343432666264656466353830346631 +62616639343265646661373062376138363232376562316134303730323431336533306565646438 +62666132613930646564393162316461643431666664356436356264633839333338356561333038 +65333466396165363030653137663939303838663030666235356639616362346330346437306133 +61653130363136643038626562653437303939393133613733326530656163323531386436656363 +37303930613063366537623435396564643431346266373363333332663632663939346131373565 +32663430643034663530643431346335623636663265316431336361363233383761333130353530 +30373566316336646536313938353663666436643532303433643139376232646566616430366330 +36646433393037373733633562306631623164356237326532653833633031633735653463343461 +65646664383531346531353133313334663130393237353436623638333731326139653363383032 +63356235646366333534366666383961636361626636616237323237346234383035623365376137 +34646162663761343662353631393131313562326138336565383764666333383731353730376135 +64656231613630646330643133373766643330656166346134363263373966363665323964396233 +64333038373565343330626633323933333239623634333436666338633539316263616438626461 +30643566656534666533373365613861383231373062613831653537623532353561633736633866 +61383532633430323330646137613633383830393136303332643531383466353039353564393232 +39326130333336663038643766303464363664333261313137663863616466666534383831613732 +35303831633232306131636137656461653361346134373935333462306232323165313265323239 +36343864306130303537386164653866656632633861363736336362656161356463386265336430 +38323739336364303061363437346137633436653566353464323462643838313631393235643635 +65626461643263656537616638623066346636663335383861376336623932343439353231343739 +64663263613862626266616362313136373630636262633036386462666632623336343431333231 +31333164356538616562353335313161353339653262616439636134383063346566663261323938 +36633631636265396230313033373230373635643165613239623532613436376464646635326535 +39333433336361343530623434356636363433653035333934653162363135626564346161346130 +65313130393639393539633534653330343935376365656162383132643635313034383538333333 +64346336346231313939396463643166363165393439626431386334393563646337383361326431 +34346462333732643365303230383937316462623566346238393431656433323535376361623561 +66333530346638623332633832623532393035343930663434343737643032303764343138326638 +33386666656162393633636437656564326336343661643137323362653437323939663033316230 +61643437633665643439663566663232643162656139383236356538663139333162383665343766 +31326539313163333034666233316634396434316535313264313430313432313938383534663333 +33653165303864353638303833646134363362306632336133306430353165363063353735336464 +31303536333964333833343566383238636233366233656531343664346331386230316365626662 +30316635353262303962316364386263636531396636306534306464336536343138663933326236 +38653563633663356339633335376361326132356334626163616361383134306239643365343562 +34323031383638313336663937663564376362663534373464386635653164636430643266343766 +62326134623638343735613562366232653137613265653836326461386539646133303638613639 +31373465353533383237393933626333383839313736636132646564666639336530386537663239 +34653533643039633337353535393832343439346536336232373837363734366430336539373732 +66646339616366313834333562303466373962656435373530666466383339653064666532663236 +64393032623234623561313766343233643232303630613165383761323435313130303361346237 +62323430353333383539613536646439336466633435383134626530326531643135326264376662 +37393832643662386137613230633964396338616638313764373464616665316531396462326431 +36666338666536613666643032376331626132316433653036656330353834323865663865366535 +36313937346136353534373631396630633432313065666639623538343332636665313236336366 +35613330336136623331346636646134316265613863323730663837303032303131376435643131 +39353965376565356136613533623235303031646331663166396631336434623561643464656263 +64376636376337336661363133663139343935613033333266663439326239646434633131316637 +38643164383136343939663332653433376362363432333265623166643831303639376637366363 +37336361336461363437646234663330393830393861373162663463366533366637646530373838 +32633764373164646339653931653066333833363534663234663864336465333164353231663032 +36333666653633643362343334303035656539636663306333613964663837366338613565626635 +34393964303866313835373936376130626665303031376637396639353734373337336534376563 +30643261663235323831366537613434376566323837633136306663663331333863643037336133 +63623035376233393431373361653964323833396362646661343235343332636435356365346330 +39663533353631643062613839643761633866373539613536326132386336373731633239323362 +31383863333330366665393666363864613632636537363436613230623030353436373061633037 +36326633356533666438353034343966366136316331376431396536393261383633373131656530 +36336364653962303730393338313435316162393561383161646232663430616239623964623431 +38383563386266343935666463636533353361623965326232663937326462366433366534306534 +66323139346238343031313762353166646235313463363435393236633337303561333930376439 +37613630386636636634396437623866326639343863353134346438663033353232646566336164 +63376535363235653362623630636661636162313462303664393032326538613139653039653065 +38626234353034643330333861316665306665313933363234623639373765303065376632643233 +39633764346132356564333236306333373664366134363037633730653732643130616665343466 +62356138656331626437666465663132663562336365353834353632383062326164356161366138 +66663532373464316264366661343863363265616438323132646361346135653063303031323833 +33306231353231316165373137616134326163313430613963653532316331323738633639356565 +38343537616330376263636165396139396365643663393637626132306564613263323362663637 +64356665646532326466393733303130386230363831666265396635363630336331326662363864 +39383432346265376137636639613162643135343433303435336436376161303766383232656166 +32323462313432643330303039373239323939626266383232393433366232636134643636646431 +37376330343461393766643664393864633265393631383566396530393364346634626535373133 +61623931386433396630616463353236393630343838383865666466383632363631373136393930 +35633934393963626263646137623262666335306233356439633334313765353161653566613833 +32346637393530313730376564656434393737383037613664666338666435373066303338623930 +63333763633533373437363666616263646564613466613830336134353933626336343036646137 +32633265636134333438386330663764633432376633383930333835363362613738353464636330 +38613135326635643436346565633833623630623736623231306237313662666666653532333234 +62383866373939326335653461653238346336636636383961613534326336653030386534653934 +39323266373664346437613362373162643532636566623563303139353430343638656130303866 +63346537323431613434643865353865633331653233343839376237646337613566663130306537 +39343735626236653636393465633038323465656661626561656438373835343361363735386638 +39623830303864636333373534396562623630653362366166386264643438373435386438383831 +34633264613230643434313037323538346661346464393362613434303135313136376131666530 +36373866313739323962333061343836363661346334383033613534393666636631363262323238 +64313330316464336533663764386666313732616338393162393434613437623730613038356663 +39383236316664313731626330323039373937366638373461373736633239666438306437666666 +66316639346432316631616261626262666238346530353563356436353332313362353431646430 +64643366656466333363313135653331303764633839316239333630353665666465393263633333 +31306663326366643032653762646464396530643662303636393432373966393934396634343032 +31346365633933313436643937626634656535323431616231393866663334303632613237633666 +65396262363665353837366166633435326133356330383066613537653736656565306434303663 +38313837303965633236366533386661366464613262326162316365626534356237663035396264 +33356165623232323664363731383962616465653230373234323961646236356132383031346465 +61396136636137303933346462356339393236313361316465346631666432626434636366396531 +35313264323766663436666430313864303065313633396631656438666564313038386161313433 +66383734373435633838666463643636376366633334626130333730366639663961666630383234 +35343964363931663431373162653333613563366462333037363838623662303837626237396132 +36366162313766366237306337333338623937363165306433616530326435363063333938393939 +61373433666233393962633263386534333735303564326435623363663334313163326239653966 +30656132353938326530306663613637373838323432323762306263623136653937383465613038 +66346661646231623161373233653337353162636334376137373138613864353036663964336362 +64303362306239633339313666303636626264353833333333613132316261333830616232333662 +64623061633466336336623734616632303265363630356665303663636334656136636339353534 +65636564636462346133666438623461653630333833643230623632333266663066376461316661 +37623637356333386638343937613361343265663033326337666138393233646639656232333864 +37343733646432353564316237363964656532386561666538633737323739383766316337386137 +65656231336463623965653232323162363866373732656534376430356432653539393939343661 +30626130336436343033386637393334363237373737613761363633316533623033343731356338 +38363231303637323861376662356564323032316130356461653466336165363363383637313862 +35666338383164666135633734386432353137363464376538663465353330376237363333653566 +32376462383732656135333861323465396664353032393432373861353836376239346338393230 +39623532336232343761303235393963333065653935616230656236646330323465303263663236 +39336233303030656161343961353566626462616333643235623032623338646163346564383034 +63333439653936653731653337396231396562383565363837643138373864613338386164346262 +31613133303833323238343665353238323335623162353362646337336335323639363730386462 +61633966313435313165643537626330663864623762633330383730613131396463313231343837 +61623965393836303230626463363438343239363562643334653733343865313334383233343932 +33313930666136613766376435636432303466626565326434636561373565306532376239643735 +63613037616534656265663536373164643866653466326361313965383635626331636463383363 +35356533393461376132626431383839393132663161373831366130663964313839613566643165 +63353139316333666337636635303236643034623830353462626535623064653065316135343564 +64363963383230313765643262396330663938366631623834376532643937316236623865613665 +34653331333336333364396330646632326266626638316566363539313532323762383034323036 +38343838616264333737376432366535643339323734316563386439313339653763343139306565 +61383538613761353733653933316634316138313565393932623638636135303962366562623231 +38346365616434663362306138373634613334613037613961383032326661356163626238666233 +31653938633938366632326439333237663539383461646236306265326536626135343161396137 +34616636663639396534363431666464306538623632623133623133643864313839653865623266 +35643865366465346532386130353565633333613061396139383265663734323139663532376638 +38623966306639363938643564343330636363346261333166613938623936303966613037663365 +62376334613363323563363332356463376532323964333338623239326663343031373635663032 +33626432356662636130363761323862616131653630396164623435333061366564626236643936 +65316562633036366233303538373530343932616533346637656464346166306438396435666533 +37313766366636336535363536333061373264663163303162666330633061356464353164636239 +39623836663066623431343835623264373239363766323338353633336338363536643462666236 +35313031633561353066306336336362643863393730613938373539363831336266353266633036 +30313039336631396664383434316462643366643532353638396339353430303063303734343830 +31323761643235663061336237333866323139613533366463353433613832316232323938323331 +39356261636231353936313033363962353232663831316634653634323965313035663537323864 +33633138383431376131616462363337313035326339646661643730666337393266316562333138 +39616636323630356631653834313466393533333761666534333239306666356166363433326434 +33643566323438333934363236313633666465663236353034653439323338613433333538336535 +30326435383362343065356463373761333565396131336562636236353863613635636436326131 +37353238656337633135643430306532393530653032663437303139323463613136656563613633 +39303463656631336231643733323439376331653732376639303231333533373264623034316435 +64386635636565333961663936356131643536636235666433356338646531306535653366363362 +64336634396666303364663033383034616639646532333238336138343032376631326139646664 +37323432613664343032313838663561373735323938666339336133653131383263366665656537 +61363139356666303438316434626536323362306539613930353263663263353933353664393666 +37663763616635306665386163333034643465376130313861633435366666653535383234363438 +34326565373461343035616162316532333739386562383466626436366434306261363931653464 +63306238663038373631363834313261323337393263323362306636663765393535303433643562 +63616336383563343732653034316133343662333733323231623064386561623535376235336432 +38646537303136653939346335653031636266343132626262643439363564613132356634666435 +34626562336630363363393261643735353263653462353336383630366335643966396434363034 +65623233346335343637633564393466616365626361613134306362663333353730336264616664 +33656361323631663639303033663632663262313461663133646663633833383833636661336463 +37633266616639663830343333336362653838316363303637333535663136336633393339666263 +37313434303638616530336366303865333038326534656135613136613930376635363062353962 +38323339356235663965383633323665313932643632363132386666363239633666313566653063 +64643433363234396563356435383335623133613364313834396236376339633738326235363737 +31303566613461386434646233306261393835613739353831306164343164616432656466636139 +62393132636465663965633432353665643535643233313738323236383166653838663338323462 +61383035636632323863333361643333363238636436616562393465326230373164366261313564 +39343861333562373762623437613334396634613462373132386564616462333638373161653737 +38313366373864333064646133393561396537353333626462666361326532323239376431326330 +30373734346566663235616235316633636263373431326166623734303734383337643562666536 +61353130646537623936646639393439306639356130373431623431643834386632616165383133 +61666165396634613763306561356533663263386639643766306166303162376366363530396361 +65356538663638633261636332386262343236663731313932656339323239343938666662343064 +66353234346530366163656437356464376136663261343938636566333865313261396164316137 +38663238656365383131363239333537646333656136656465636464656530306137363436636231 +39366339663630653461383461323234363936346230643266393833356432396661356263333238 +38656530306133363733336335386238626132356237313334633634393634333937643631623963 +36326364306664613832363734336463303635373262653632356263336665353739373231316664 +34633036383439393339363364346165376261613339356236323330306432616665343265363564 +66666137346231383465353666633835393737313261643935656530356162613538363864343830 +35323865613563366130323132326238346338303761353433656532643131333962373066336233 +34303734396235303137616362653037333138306132393537646531616635636561343831366531 +36333932323234643237346266323466303736663665363362373465333139393934623939313535 +34316433323136316533313663393238663530386565376131366631393031643362346239316430 +64303636343765326465616464663234373137393137666635306261653466323733303962393461 +61356438316631333231636465653933343264356230313937636431336261306464316532343335 +32383732363866383531373136353331623566636332643332636237653737313064393566373837 +37333031643034343637333832363931343938303764313836373230353931613535313765393065 +30623234653039316461626635366337633265623236333963323934313065623732353362303338 +64663130383030613536383739646562636662656234363036613162666461373834333530363363 +62383166353333663232363737653639316235303535616536383962653039386464303435356538 +61653166633162616563336663323830383732666264393262323139303936626464373132303738 +30343061313535363838653930306164303438353736393862643461363333396539326633386236 +63623666343330363166373037636332313966303565303061663361386239353962666164383435 +64626663383735393431623436323862656262396461393663616530353132646234353838633537 +66313437306330653963613663373235383338383439636563643333323265626132326639386639 +38663762326666343935393664643761393236323266336265393033643734333439353538613062 +38626438663431366464343937316165313937323732643061643334643437353764333231646362 +65393566663530383133393430383237633638363363313963383230663434353635353364333830 +30306535353962633865346131333035353638336539336635643238383762303563323663616164 +63356630393664653637613935363536656239616239316337353730666632633234613331336136 +34383036616432306533663164316163303332313239646662656536333564643539343533383939 +32303935313565626661393334303564633136323164623965663364323966346636396532336535 +38373232616265346131633930333134666333663961383063666237653462623635323537393837 +30373438616561353430633139323163353138363735636530643630346566633631383931363638 +66633266643762626530613238343663326236363134316337653861376561383631663937373833 +32663063356164333731343263393837386332623665313135396665373436323366613331363333 +65393533353933643134643362366330653261313064663963336539653134396437383430313666 +33663234616461626630646563616561313139396630646461386338623266666434376335306435 +36643430663131373064313464633733356131386338643439653032353436656661356366626533 +35343434396361303634636438633036386563383664646561663566613339323865363136333230 +37616265613030633061376639386461383062353135666534346632353064353231 diff --git a/ansible/host_vars/jar_lid_staging_local/main.yml b/ansible/host_vars/jar_lid_staging_local/main.yml new file mode 100644 index 0000000..7885123 --- /dev/null +++ b/ansible/host_vars/jar_lid_staging_local/main.yml @@ -0,0 +1,28 @@ +--- +# Jar Lid Archive STAGING. Secrets are in vault.yml beside this file. +# +# These live here rather than in the role defaults because three plays read +# them: the deploy on this box, the Caddy vhost on the edge host (port, domain) +# and the backup (retention). A role default cannot reach a second play. + +# Names this deployment everywhere it must be unique across hosts: the Caddy site +# file, the Gatus probe (probe_), the backup artefacts and the +# backup-dump/backup-store endpoints. +jar_lid_archive_instance: jar-lid-archive-staging + +# Fixed by the image's CMD (gunicorn --bind 0.0.0.0:8000); changing this moves +# what Caddy and the probe call, not where gunicorn listens. +jar_lid_archive_port: 8000 +jar_lid_archive_domain: "{{ subdomains.jar_lid_archive_staging }}.{{ root_domain }}" + +# Staging data is disposable: three days locally and on small-backups-box. +jar_lid_archive_backup_retention_days: 3 + +jar_lid_archive_seed_users: + - {username: alice, email: alice@example.com, password: "{{ jar_lid_archive_seed_password_alice }}"} + - {username: bob, email: bob@example.com, password: "{{ jar_lid_archive_seed_password_bob }}"} + - {username: admin, email: admin@example.com, password: "{{ jar_lid_archive_seed_password_admin }}", is_superuser: true} + +# No systemd units of ours run here - the stack is docker compose - so infra/401 +# has nothing to watch. The probe in the deploy playbook calls /health/, which +# checks the database and the bucket as well as gunicorn. diff --git a/ansible/host_vars/jar_lid_staging_local/vault.yml b/ansible/host_vars/jar_lid_staging_local/vault.yml new file mode 100644 index 0000000..0de8609 --- /dev/null +++ b/ansible/host_vars/jar_lid_staging_local/vault.yml @@ -0,0 +1,34 @@ +$ANSIBLE_VAULT;1.1;AES256 +37383535306133363031316535656461613038653562373839363432623163356234303061643566 +6533333535613064653336313664643063326562666639640a373530386539316363346332663137 +31303230323065623733663666383039346337343239646465393166663239643836356639373765 +3564313065363266360a643239333330616463616533623435656164386264303634356134303831 +36356131323637363734316161383065653663623732386638363165633265356362386538643039 +64636464323730316638316430643339356662353736366332663738656566663137366166393735 +34633430363931323330393234336661383532653366633735313365363562396137356438353330 +65313737313131326361326631363864396536343633623963303632623465363964373964393430 +61323161663939346435386435616364653336646165356232346335353235356339656336306138 +65613636353636623562613362633761333637613066663838396638613161663532393030303035 +35316464356539666631636265646433353537396564366538643962373364383730626164316265 +63393034663835346362636564316135326430633963393139646365333666663630336462323361 +31353830633335366635366435643864313665366563396134313938666537663232343162383366 +34636231636261663739383366363962366438323962393732393864623238343961363035333866 +61386630396334626564306536336561333265383464623730386263373132613438396332356631 +30386435333739653637633032313966626437303634613838653735373666343565306663376230 +61303734623865353835626464613033346437623234626136353664303434386262346634356463 +34396433353431393935386136333331643539633138636135353339323930386634393639666665 +36643462653061363062383039313936303062393364353366623965313530303031306336396336 +65643663343163636338393666386134366337633064376530303463393737396165663930383562 +64316363663636383764656139366433393766613262613433623637393538633130616232373136 +31376636633666626365376533356262393065656539613266326465643663383439373661393631 +30636136363066653936653837376634356333353036633165316662383337636435323162316430 +30316331353361396132653233623034376539323064363135306262376638353664643163653334 +37643638323038663531376336313064383238366430663363353733356530393661646138303230 +30633862626166653362346632626261636531373130353034656532386230616330643134383631 +63396138326336626431656130306531626539323934386332306361396239376466623866613364 +36666632316331326361373136326465323365653034363337313135656262623465306439646362 +30636639663163643430363165316231366232623332393635623461306339313762323265623732 +30633464343235323566356434376535313238663533633630356163336532303765633366316166 +38633462303762323930393337306265393033653061616337646163333335333737666332666632 +35613132646636333335313263343136323337616662623465653939336637653134383766336265 +35633638663636353537333238643236623033636163333164663062353863643164 diff --git a/ansible/infra/402_public_monitoring.yml b/ansible/infra/402_public_monitoring.yml index 61968be..5aec9c0 100644 --- a/ansible/infra/402_public_monitoring.yml +++ b/ansible/infra/402_public_monitoring.yml @@ -37,6 +37,7 @@ - {sub: "{{ subdomains.monero }}", host: vipy, interval: '24h'} - {sub: "{{ subdomains.monero_explorer }}", host: vipy, interval: '24h'} - {sub: "{{ subdomains.music }}", host: vipy} + - {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy} # A public resolver on purpose: this must test what the internet sees, not # what a local cache or the tailnet's MagicDNS happens to answer. @@ -75,6 +76,10 @@ # API the mobile clients speak cannot carry a proxy credential, so its # own login is the gate. See roles/navidrome/README.md. - {name: music, sub: "{{ subdomains.music }}", path: "/app/", status: 200} + # /health/ through the whole public path: DNS, vipy's Caddy and cert, the + # tailnet hop, gunicorn, and the database and bucket behind it (503 if + # either is down). "/" would only prove the login redirect works. + - {name: jars-staging, sub: "{{ subdomains.jar_lid_archive_staging }}", path: "/health/", status: 200} # Ports published from the edge host by socket_proxy. public_tcp: diff --git a/ansible/infra/403_service_probe_registration.yml b/ansible/infra/403_service_probe_registration.yml index 1a53a63..e1ed107 100644 --- a/ansible/infra/403_service_probe_registration.yml +++ b/ansible/infra/403_service_probe_registration.yml @@ -55,6 +55,7 @@ - {name: monerod, host: monerod_box_local, heartbeat: '26h'} - {name: xmrblocks, host: monerod_box_local, heartbeat: '26h'} - {name: navidrome, host: music_box_local} + - {name: jar-lid-archive-staging, host: jar_lid_staging_local} tasks: - name: Build the probe endpoint list diff --git a/ansible/inventory.ini b/ansible/inventory.ini index 9489607..f5964ab 100644 --- a/ansible/inventory.ini +++ b/ansible/inventory.ini @@ -21,6 +21,10 @@ monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=cou # key was still in known_hosts and SSH refused the connection, which is the # designed behaviour described in group_vars/nodito_vms.yml. music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua +# lan_ip read from the Proxmox guest agent for VM 3300 (qm guest cmd +# network-get-interfaces), and the host key checked against the one the agent +# reads off the disk before it went into known_hosts. +jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1.151 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua # Local connection to laptop: this assumes you're running ansible commands from your personal laptop [lapy] @@ -63,6 +67,10 @@ memos_box_local [music] music_box_local +# Staging only for now; a production host joins this group with its own host_vars. +[jar_lid_archive] +jar_lid_staging_local + [ci_runner] forgejo_runner_local diff --git a/ansible/playbooks/backups.yml b/ansible/playbooks/backups.yml index 0be3807..875f10a 100644 --- a/ansible/playbooks/backups.yml +++ b/ansible/playbooks/backups.yml @@ -30,6 +30,10 @@ - name: forgejo source: "backup-pull@prd-vipy:/opt/backups/forgejo/" retention_days: 14 + # Staging: disposable data, so the long tail is as short as the local one. + - name: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}" + source: "backup-pull@{{ hostvars['jar_lid_staging_local'].ansible_host }}:/opt/backups/{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}/" + retention_days: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_backup_retention_days }}" # ───────────────────────────────────────────────────────────────────────────── # Register the backup checks with Gatus. @@ -62,7 +66,8 @@ - {name: vaultwarden, host: vipy} - {name: lnbits, host: vipy} - {name: forgejo, host: vipy} - store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo] + - {name: jar-lid-archive-staging, host: jar_lid_staging_local} + store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo, jar-lid-archive-staging] tasks: # 26h, not 7h: the DUMP is genuinely daily, so the window cannot be tighter diff --git a/ansible/roles/jar_lid_archive/README.md b/ansible/roles/jar_lid_archive/README.md new file mode 100644 index 0000000..8f2e641 --- /dev/null +++ b/ansible/roles/jar_lid_archive/README.md @@ -0,0 +1,121 @@ +# `jar_lid_archive` + +Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive) +as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and +SeaweedFS as the S3 photo store (`storage`). The same three services as the app +repo's own `compose.yaml`, so staging rehearses what dev and CI run. + +Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to +every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local` +(jar-lid-archive-staging-box, nodito VM 3300), public at +`jars-staging.contrapeso.xyz` through vipy's Caddy. + +Per-host values are in `host_vars//main.yml`, and secrets are in `vault.yml` +beside it. A production host would be a second inventory entry with its own +`jar_lid_archive_instance`, domain and vault, not a second playbook. + +## Updates: follows `:master` + +CI pushes `:master` on every green build of master. Every run of the playbook +runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only +if the digest moved. There is no watchtower, so **a merge reaches staging when +the playbook runs**, not on its own. The run ends by printing `/health/`, which +includes the deployed version. + +`up --wait` only returns once every container's healthcheck passes. `web`'s is +`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means +the app actually serves. + +## Networking: why `web` is on the host network + +Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT` +rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well +as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids +that, but at boot Docker races tailscaled for the address and loses. + +So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host +port: + +- **LAN:** UFW default-deny blocks it. +- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW + (checked on mempool-box), so the headscale ACL is the real control. It grants + `tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`. +- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches + them from the host network. + +This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto` +(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only +thing that can reach gunicorn. Caddy overwrites whatever value the client sent. + +## Secrets and seed users + +The Django secret key, the Postgres password, the S3 secret and the three seed +passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They +reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose +interpolates. `docker-compose.yml` itself has no secrets. + +The seed users are alice, bob and admin (superuser), with the fixture's usernames +and **vault** passwords. The fixture's `*-test-pass` passwords are published in +the app repo and this site is public. `seed_users` runs on every deploy and +resets each listed user's password, so rotating one means editing the vault and +re-running the playbook. + +```sh +ansible-vault view host_vars/jar_lid_staging_local/vault.yml # from ansible/ +``` + +## Backups + +`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses +`backup_source`. At 02:20 it stops `web` only, and +`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with: + +- `db.dump`: `pg_dump -Fc` +- `seaweedfs/`: the photo store's data directory, as files + +`web` is the only writer to the database and the photo store. With it stopped, +both are quiet and are captured at the same instant. `storage` is deliberately +left running: on this disk it can take minutes to restart, and the first test run +that stopped it left the app down for 26 minutes. Uploads fail while `web` is +down. + +The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally, +and small-backups-box keeps three as well. Staging data is disposable. + +### Restore + +```sh +age -d -i ~/.age/counterweight_age jar-lid-archive-staging_.tar.age | tar -xf - -C /tmp/restore +# on the box, stack down: +docker compose -f /opt/jar-lid-archive/docker-compose.yml down +rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/ +docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db +docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \ + pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump +docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait +``` + +## Monitoring + +| What | Where | +|---|---| +| Host liveness and disk | infra/400 (automatic for `managed`) | +| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) | +| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) | +| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` | + +infra/401 has nothing to watch here, because no systemd units of ours run on +this box. + +## Size and disk + +The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's +random reads keep that mirror close to saturated, and this box has measured +**~96% "full" I/O pressure** with nothing of its own running. Everything is slow +here, but especially container starts. The healthcheck `start_period`s are sized +for that. Moving the disk to `local-zfs` (NVMe) would fix it. + + +1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian, +the images, the photos and three days of local backups, and each backup is +about the size of the whole photo store. infra/400's disk check alarms at 85%. diff --git a/ansible/roles/jar_lid_archive/defaults/main.yml b/ansible/roles/jar_lid_archive/defaults/main.yml new file mode 100644 index 0000000..da488d2 --- /dev/null +++ b/ansible/roles/jar_lid_archive/defaults/main.yml @@ -0,0 +1,57 @@ +--- +# Jar Lid Archive: Django (gunicorn) + Postgres + SeaweedFS (S3) as one compose +# stack. Source and CI live in forgejo.contrapeso.xyz/counterweight/jar-lid-archive. +# +# Per-deployment values (domain, port, retention) live in the host's host_vars, +# because the Caddy play on the edge host and the backup plays read them too and +# a role default cannot reach a second play. + +jar_lid_archive_dir: /opt/jar-lid-archive + +# ── Images ─────────────────────────────────────────────────────────────────── +# The app follows a moving tag on purpose: every run pulls it, and compose +# recreates `web` only when the digest actually changed. CI pushes :master on +# every green build of master, plus :. +jar_lid_archive_image: forgejo.contrapeso.xyz/counterweight/jar-lid-archive +jar_lid_archive_image_tag: master + +# Pinned to what the app's own compose.yaml and CI run, so staging rehearses the +# same stack. Bump them together with the app repo, not separately. +jar_lid_archive_postgres_image: "postgres:18" +jar_lid_archive_seaweedfs_image: "chrislusf/seaweedfs:4.47" + +# Registry login. Empty = pull anonymously, which works today: the package is +# owner-scoped in Forgejo and public even though the repository is private. +jar_lid_archive_registry: forgejo.contrapeso.xyz +jar_lid_archive_registry_user: "" +jar_lid_archive_registry_token: "" + +# ── Networking ─────────────────────────────────────────────────────────────── +# gunicorn listens on the HOST network on this port. See the README for why host +# networking rather than a published port: Docker's published ports bypass UFW. +jar_lid_archive_port: 8000 +# Loopback-only ports for the two backing services, so the web container (on the +# host network) can reach them and nothing else can. +jar_lid_archive_postgres_port: 5432 +jar_lid_archive_s3_port: 8333 + +jar_lid_archive_domain: "" # e.g. jars-staging.contrapeso.xyz + +# ── App settings ───────────────────────────────────────────────────────────── +jar_lid_archive_postgres_db: jarlid +jar_lid_archive_postgres_user: jarlid +jar_lid_archive_s3_access_key: jarlid +jar_lid_archive_s3_bucket: photos + +# Secrets: pass from the vault. The assert in tasks/deploy.yml refuses empties. +jar_lid_archive_django_secret_key: "" +jar_lid_archive_postgres_password: "" +jar_lid_archive_s3_secret_key: "" + +# ── Seed users ─────────────────────────────────────────────────────────────── +# Loaded with `manage.py seed_users -` on every run. That command RESETS each +# listed user's password and flags, so the vault stays the source of truth. +# Same usernames as the app's tests/fixtures/users.json, never its passwords: +# this deployment is reachable from the internet. +# - {username: alice, password: "...", email: "...", is_superuser: false} +jar_lid_archive_seed_users: [] diff --git a/ansible/roles/jar_lid_archive/tasks/deploy.yml b/ansible/roles/jar_lid_archive/tasks/deploy.yml new file mode 100644 index 0000000..4a6e783 --- /dev/null +++ b/ansible/roles/jar_lid_archive/tasks/deploy.yml @@ -0,0 +1,100 @@ +--- +- name: Assert the deployment is fully specified + ansible.builtin.assert: + that: + - jar_lid_archive_domain | length > 0 + - jar_lid_archive_django_secret_key | length >= 50 + - jar_lid_archive_postgres_password | length > 0 + - jar_lid_archive_s3_secret_key | length > 0 + fail_msg: >- + jar_lid_archive needs a domain and its three secrets from the vault. The + app falls back to development defaults for every one of them, so an empty + value here would not fail - it would deploy with a published secret key. + no_log: true + +- name: Create the deployment directory + ansible.builtin.file: + path: "{{ jar_lid_archive_dir }}" + state: directory + owner: root + group: root + mode: '0755' + +# Existence only, no owner: the postgres and seaweedfs entrypoints chown their +# own data directories to the uid inside the container. Declaring an owner here +# would fight them on every run - the mempool MariaDB lesson. +- name: Ensure the data directories exist + ansible.builtin.file: + path: "{{ jar_lid_archive_dir }}/{{ item }}" + state: directory + loop: + - postgres + - seaweedfs + +# diff: false, because --diff prints rendered content and this is all secrets. +- name: Write the compose environment file + ansible.builtin.template: + src: env.j2 + dest: "{{ jar_lid_archive_dir }}/.env" + owner: root + group: root + mode: '0600' + diff: false + +- name: Write docker-compose.yml + ansible.builtin.template: + src: docker-compose.yml.j2 + dest: "{{ jar_lid_archive_dir }}/docker-compose.yml" + owner: root + group: root + mode: '0644' + +- name: Log in to the container registry + ansible.builtin.command: + cmd: "docker login {{ jar_lid_archive_registry }} -u {{ jar_lid_archive_registry_user }} --password-stdin" + stdin: "{{ jar_lid_archive_registry_token }}" + when: jar_lid_archive_registry_token | length > 0 + changed_when: false + no_log: true + +# Pulled on every run: staging follows :master. `up` below recreates `web` only +# when the pulled digest differs from the running one. +- name: Pull images + ansible.builtin.command: docker compose pull --quiet + args: + chdir: "{{ jar_lid_archive_dir }}" + changed_when: false + +# --wait blocks until every service's healthcheck passes, and `web`'s is +# /health/, which itself checks the database and the bucket - so this task +# succeeding means the app actually serves, not merely that containers exist. +- name: Start the stack and wait for it to be healthy + ansible.builtin.command: docker compose up -d --remove-orphans --wait --wait-timeout 300 + args: + chdir: "{{ jar_lid_archive_dir }}" + register: jar_lid_archive_up + changed_when: jar_lid_archive_up.stderr is search('Created|Recreated|Started|Removed') + +- name: Read the deployed version + ansible.builtin.uri: + url: "http://127.0.0.1:{{ jar_lid_archive_port }}/health/" + return_content: true + register: jar_lid_archive_health + # Under --check the stack was never started, so there is nothing to ask. + when: not ansible_check_mode + +- name: Show the deployed version + ansible.builtin.debug: + msg: "{{ jar_lid_archive_health.json }}" + when: not ansible_check_mode + +# Used by the backup (services/jar-lid-archive/setup_backup_jar_lid_archive.yml). +# Lives here because what to dump is a fact about this stack's layout. +- name: Install the dump script + ansible.builtin.template: + src: dump.sh.j2 + dest: /usr/local/bin/jar-lid-archive-dump.sh + owner: root + group: root + mode: '0755' + validate: "bash -n %s" diff --git a/ansible/roles/jar_lid_archive/tasks/docker.yml b/ansible/roles/jar_lid_archive/tasks/docker.yml new file mode 100644 index 0000000..2eee8dc --- /dev/null +++ b/ansible/roles/jar_lid_archive/tasks/docker.yml @@ -0,0 +1,74 @@ +--- +# Same Docker CE install as roles/mempool/tasks/docker.yml. infra/910_docker +# targets every managed host, so each Docker role installs its own engine. +- name: Remove old Docker-related packages + apt: + name: + - docker.io + - docker-doc + - docker-compose + - podman-docker + - containerd + - runc + state: absent + purge: yes + autoremove: yes + +- name: Update apt cache + apt: + update_cache: yes + +- name: Install prerequisites + apt: + name: + - ca-certificates + - curl + state: present + +- name: Create directory for Docker GPG key + file: + path: /etc/apt/keyrings + state: directory + mode: '0755' + +- name: Download Docker GPG key + get_url: + url: https://download.docker.com/linux/debian/gpg + dest: /etc/apt/keyrings/docker.asc + mode: '0644' + +- name: Get Debian architecture + command: dpkg --print-architecture + register: deb_arch + changed_when: false + +- name: Add Docker repository + apt_repository: + repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable" + filename: docker + state: present + update_cache: yes + +- name: Install Docker packages + apt: + name: + - docker-ce + - docker-ce-cli + - containerd.io + - docker-buildx-plugin + - docker-compose-plugin + state: present + update_cache: yes + +- name: Ensure Docker is started and enabled + systemd: + name: docker + enabled: yes + state: started + +- name: Add user to docker group + user: + name: "{{ ansible_user }}" + groups: docker + append: yes + diff --git a/ansible/roles/jar_lid_archive/tasks/main.yml b/ansible/roles/jar_lid_archive/tasks/main.yml new file mode 100644 index 0000000..c16a270 --- /dev/null +++ b/ansible/roles/jar_lid_archive/tasks/main.yml @@ -0,0 +1,6 @@ +--- +# import_tasks, not include_tasks: static imports stay visible to --list-tasks, +# which is the main verification tool in this repo. +- ansible.builtin.import_tasks: docker.yml +- ansible.builtin.import_tasks: deploy.yml +- ansible.builtin.import_tasks: seed.yml diff --git a/ansible/roles/jar_lid_archive/tasks/seed.yml b/ansible/roles/jar_lid_archive/tasks/seed.yml new file mode 100644 index 0000000..6b87c31 --- /dev/null +++ b/ansible/roles/jar_lid_archive/tasks/seed.yml @@ -0,0 +1,13 @@ +--- +# seed_users is idempotent but RESETS each listed user's password and flags, so +# re-running it is how a rotated vault password reaches the app. It reports +# "N created, M updated"; only a creation counts as a change. +- name: Seed users + ansible.builtin.command: + cmd: docker compose exec -T web python manage.py seed_users - + chdir: "{{ jar_lid_archive_dir }}" + stdin: "{{ jar_lid_archive_seed_users | to_json }}" + register: jar_lid_archive_seed + changed_when: "jar_lid_archive_seed.stdout is not search('Seeded users: 0 created')" + when: jar_lid_archive_seed_users | length > 0 + no_log: true diff --git a/ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2 b/ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2 new file mode 100644 index 0000000..cc634f7 --- /dev/null +++ b/ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2 @@ -0,0 +1,123 @@ +# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host. +# +# Same three services as the app repo's compose.yaml, with three differences: +# - `web` runs the CI-built image instead of `build: .` +# - `web` is on the HOST network, so UFW and the tailnet ACL decide who reaches +# gunicorn. A published port would be DNATed by Docker straight past UFW and +# answer on the LAN too. See the role README. +# - data is bind-mounted under {{ jar_lid_archive_dir }} rather than in named +# volumes, so the backup tars a path it can name. +# +# Secrets are interpolated from .env (0600) beside this file; this file has none. +name: jar-lid-archive + +x-logging: &logging + # Docker's default json-file driver never rotates, on a 10 GB disk. + driver: json-file + options: + max-size: "10m" + max-file: "3" + +x-postgres-env: &postgres-env + POSTGRES_DB: "{{ jar_lid_archive_postgres_db }}" + POSTGRES_USER: "{{ jar_lid_archive_postgres_user }}" + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?} + +x-s3-env: &s3-env + S3_ACCESS_KEY: "{{ jar_lid_archive_s3_access_key }}" + S3_SECRET_KEY: ${S3_SECRET_KEY:?} + +services: + web: + image: "{{ jar_lid_archive_image }}:{{ jar_lid_archive_image_tag }}" + # gunicorn binds 0.0.0.0:{{ jar_lid_archive_port }} - the port is fixed by the image's CMD. + network_mode: host + environment: + <<: [*postgres-env, *s3-env] + POSTGRES_HOST: 127.0.0.1 + POSTGRES_PORT: "{{ jar_lid_archive_postgres_port }}" + S3_ENDPOINT_URL: "http://127.0.0.1:{{ jar_lid_archive_s3_port }}" + S3_BUCKET: "{{ jar_lid_archive_s3_bucket }}" + DJANGO_DEBUG: "0" + DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY:?} + # 127.0.0.1 for the container healthcheck and the probe, which both call + # /health/ directly rather than through Caddy. + DJANGO_ALLOWED_HOSTS: "{{ jar_lid_archive_domain }},127.0.0.1,localhost" + # Caddy terminates TLS and forwards plain HTTP with X-Forwarded-Proto. The + # app must trust that header or every POST fails CSRF and share links come + # out http://. Safe only because nothing but Caddy can reach gunicorn. + DJANGO_BEHIND_TLS_PROXY: "1" + DJANGO_CSRF_TRUSTED_ORIGINS: "https://{{ jar_lid_archive_domain }}" + depends_on: + db: + condition: service_healthy + storage: + condition: service_healthy + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:{{ jar_lid_archive_port }}/health/')"] + interval: 30s + timeout: 5s + retries: 5 + start_period: 180s + logging: *logging + restart: unless-stopped + + db: + image: "{{ jar_lid_archive_postgres_image }}" + environment: *postgres-env + ports: + - "127.0.0.1:{{ jar_lid_archive_postgres_port }}:5432" + volumes: + - ./postgres:/var/lib/postgresql + healthcheck: + test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"] + interval: 5s + timeout: 3s + retries: 15 + start_period: 120s + logging: *logging + restart: unless-stopped + + storage: + image: "{{ jar_lid_archive_seaweedfs_image }}" + environment: *s3-env + # Copied from the app's compose.yaml: write the S3 identity from the + # environment, then hand over to the image's entrypoint, which drops root + # and runs master + volume + filer + S3 gateway. + # + # The `rm -f` is ours, and load-bearing. On a RESTART of the same container + # (a reboot, a backup's stop/start) /tmp/s3.json survives, owned by seaweed + # from the chown below. Root cannot then re-create it: /tmp is sticky and + # the kernel's protected_regular refuses O_CREAT on another user's file + # there. The result was "can't create /tmp/s3.json: Permission denied", an + # unhealthy storage container, and web never starting - found on the first + # backup run. + entrypoint: ["sh", "-c"] + command: + - | + rm -f /tmp/s3.json + umask 077 + cat > /tmp/s3.json < pg_restore -d jarlid db.dump +# seaweedfs/ the photo store, as files +# +# Called with `web` stopped (the backup playbook arranges it). web is the only +# writer to db and storage, so both are quiescent and agree with each other: no +# row points at a photo the tar does not have. storage keeps running; with no +# writes arriving its volume files do not change under tar. +# +# The pg_dump lands in a temp dir on the same disk as the live database, which +# is already plaintext; it is gone before this script exits. +# +# tar, not tar.gz: the bulk is JPEGs, which do not compress, and gzip on one +# core would only make the window longer. +set -euo pipefail + +cd {{ jar_lid_archive_dir }} +tmp=$(mktemp -d {{ jar_lid_archive_dir }}/.dump.XXXXXX) +trap 'rm -rf "$tmp"' EXIT + +docker compose exec -T db pg_dump -U {{ jar_lid_archive_postgres_user }} -d {{ jar_lid_archive_postgres_db }} -Fc > "$tmp/db.dump" +tar -cf - -C "$tmp" db.dump -C {{ jar_lid_archive_dir }} seaweedfs diff --git a/ansible/roles/jar_lid_archive/templates/env.j2 b/ansible/roles/jar_lid_archive/templates/env.j2 new file mode 100644 index 0000000..7027cae --- /dev/null +++ b/ansible/roles/jar_lid_archive/templates/env.j2 @@ -0,0 +1,5 @@ +# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host. +# Read by docker compose to interpolate docker-compose.yml. 0600: secrets. +POSTGRES_PASSWORD={{ jar_lid_archive_postgres_password }} +S3_SECRET_KEY={{ jar_lid_archive_s3_secret_key }} +DJANGO_SECRET_KEY={{ jar_lid_archive_django_secret_key }} diff --git a/ansible/services/headscale/templates/acl.json.j2 b/ansible/services/headscale/templates/acl.json.j2 index b120114..bc60624 100644 --- a/ansible/services/headscale/templates/acl.json.j2 +++ b/ansible/services/headscale/templates/acl.json.j2 @@ -43,6 +43,7 @@ "tag:mempool-box:8080", "tag:memos-box:5230", "tag:music-box:4533", + "tag:jar-lid-archive-staging-box:8000", "tag:monerod-box:18089", "tag:monerod-box:8081", "tag:arbret-staging-box:80", @@ -82,7 +83,8 @@ "tag:prd-arbret:22", "tag:prd-vipy:22", "tag:memos-box:22", - "tag:stalwart-box:22" + "tag:stalwart-box:22", + "tag:jar-lid-archive-staging-box:22" ], "src": [ "tag:small-backups-box" @@ -102,6 +104,9 @@ "tag:fulcrum-box": [ "{{ headscale_namespace }}@" ], + "tag:jar-lid-archive-staging-box": [ + "{{ headscale_namespace }}@" + ], "tag:knots-box": [ "{{ headscale_namespace }}@" ], diff --git a/ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml b/ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml new file mode 100644 index 0000000..4e9a9d6 --- /dev/null +++ b/ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml @@ -0,0 +1,51 @@ +--- +# Jar Lid Archive on its own box, published from the edge host. +# +# See roles/jar_lid_archive/README.md. Per-host values (domain, port, seed users, +# secrets) are in host_vars//, so a production host is a new inventory +# entry in [jar_lid_archive], not a new playbook. +- name: Deploy Jar Lid Archive + hosts: jar_lid_archive + become: yes + roles: + - jar_lid_archive + # "Does the app answer", as opposed to "are the containers up". /health/ + # runs SELECT 1 and touches the bucket, and returns 503 if either fails. + - role: healthcheck + vars: + healthcheck_name: jar-lid-archive + healthcheck_description: "Jar Lid Archive /health/ (app, database and photo store)" + healthcheck_command: "curl -fsS --max-time 10 -o /dev/null http://127.0.0.1:{{ jar_lid_archive_port }}/health/" + healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_{{ jar_lid_archive_instance }}/external" + healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" + +- name: Publish Jar Lid Archive from the edge host + hosts: edge + become: yes + tasks: + # No basic auth: the one-off links (/links/...) are opened by people who have + # no account and no proxy credential. The app's own login guards the rest. + # + # Caddy sets X-Forwarded-Proto: https and overwrites any value the client + # sent, which is what lets the app trust it (DJANGO_BEHIND_TLS_PROXY). + # + # No request size limit is set, deliberately: Caddy has none by default, and + # phones upload multi-megabyte photos. + - name: Expose Jar Lid Archive over HTTPS + ansible.builtin.include_role: + name: caddy_site + vars: + caddy_site_name: "{{ hostvars[jla_host].jar_lid_archive_instance }}" + caddy_site_domain: "{{ hostvars[jla_host].jar_lid_archive_domain }}" + # Tailscale MagicDNS: the box has no public address. + caddy_site_upstream: "{{ hostvars[jla_host].ansible_host }}:{{ hostvars[jla_host].jar_lid_archive_port }}" + caddy_site_resolvers: "100.100.100.100" + loop: "{{ groups['jar_lid_archive'] }}" + # Not `item`: the role's own tasks loop, and would shadow it. + loop_control: + loop_var: jla_host + + # The per-site validate cannot catch a conflict between sites. + - name: Validate the whole Caddyfile + ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile + changed_when: false diff --git a/ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml b/ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml new file mode 100644 index 0000000..d0a0147 --- /dev/null +++ b/ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml @@ -0,0 +1,34 @@ +--- +# Jar Lid Archive backup: the database and the photo store, as ONE artefact, +# dumped and age-encrypted on the box itself. small-backups-box pulls it. +# +# Only `web` is stopped. It is the only writer to both the database and the +# photo store, so with it down both are quiescent and are captured at the same +# instant - no row points at a photo the tar lacks. db stays up for pg_dump. +# +# storage is deliberately NOT stopped, though an earlier version did: on this +# VM's contended HDD-backed disk SeaweedFS can take minutes to come back, and the +# first test run left the app down for 26 minutes waiting on it. +- name: Configure the Jar Lid Archive backup on its own host + hosts: jar_lid_archive + become: yes + + tasks: + - name: Ensure Jar Lid Archive dumps itself, encrypted, on a timer + ansible.builtin.include_role: + name: backup_source + vars: + backup_source_name: "{{ jar_lid_archive_instance }}" + backup_source_description: "Jar Lid Archive ({{ jar_lid_archive_instance }})" + backup_source_dump_command: /usr/local/bin/jar-lid-archive-dump.sh + # Uncompressed: the bulk is JPEGs. See roles/jar_lid_archive/templates/dump.sh.j2. + backup_source_artifact_suffix: "tar.age" + backup_source_stop_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml stop web" + # `up -d --wait`, not `start`: it also repairs a db or storage that died + # meanwhile, and only returns once /health/ passes. + backup_source_start_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait --wait-timeout 300" + backup_source_retention_days: "{{ jar_lid_archive_backup_retention_days }}" + backup_source_on_calendar: "*-*-* 02:20:00" + # Reported to Gatus as backup-dump_. + backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_{{ jar_lid_archive_instance }}/external" + backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" diff --git a/ansible/site.yml b/ansible/site.yml index 4dc5680..40f410c 100644 --- a/ansible/site.yml +++ b/ansible/site.yml @@ -57,6 +57,7 @@ - import_playbook: services/xmrblocks/deploy_xmrblocks_playbook.yml - import_playbook: services/memos/deploy_memos_playbook.yml - import_playbook: services/navidrome/deploy_navidrome_playbook.yml +- import_playbook: services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml - import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml - import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml - import_playbook: services/headscale/deploy_headscale_playbook.yml @@ -72,6 +73,7 @@ - import_playbook: services/forgejo/setup_backup_forgejo.yml - import_playbook: services/lnbits/setup_backup_lnbits.yml - import_playbook: services/memos/setup_backup_memos.yml +- import_playbook: services/jar-lid-archive/setup_backup_jar_lid_archive.yml - import_playbook: playbooks/backups.yml # Deliberately not here. Every playbook in the repo is either imported above or diff --git a/tofu/nodito/vms.auto.tfvars b/tofu/nodito/vms.auto.tfvars index 5db24a9..30596c7 100644 --- a/tofu/nodito/vms.auto.tfvars +++ b/tofu/nodito/vms.auto.tfvars @@ -90,6 +90,14 @@ vms = { disk_size_gb = 20 ipconfig0 = "ip=dhcp" }, + "jar-lid-archive-staging-box" = { + vmid = 3300 + name = "jar-lid-archive-staging-box" + cores = 1 + memory_mb = 1024 + disk_size_gb = 10 + ipconfig0 = "ip=dhcp" + }, # disk_size_gb here is the music library's ceiling, not just the OS: the # library lives on the root disk at /srv/music. ~45 GB is usable after # Debian, so roughly 300-450 albums in FLAC. Growing it means a Proxmox From 71dbfbe5ff036ddcc240d6647985e158ea39dee8 Mon Sep 17 00:00:00 2001 From: counterweight Date: Sat, 17 Oct 2026 17:30:57 +0200 Subject: [PATCH 02/15] monero: find the chain disk by its label, and a blank one by Proxmox slot After the chain disk moved to the SSD, a reboot on 2026-10-04 swapped sda and sdb, and monero_data_device: /dev/sdb then named the ROOT disk. The root-disk guard caught it, but a routine run should never have been reading a device name at all. Every run now looks for the filesystem labelled monero-data first. If it is there, monero_data_device is never consulted and the mount goes by that filesystem's UUID. Only a blank disk is found by name, and the name is now /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1: Proxmox passes the slot in as the serial, so it is fixed before boot and survives `qm disk move`. The by-id path is resolved with readlink before the root-disk guard compares it, since a symlink path is never a prefix of /dev/sdb1 and the guard would otherwise always pass. The UUID assert now also fails on two disks carrying the same label. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/host_vars/monerod_box_local/main.yml | 10 +- ansible/roles/monero/README.md | 13 +- ansible/roles/monero/defaults/main.yml | 23 +++- ansible/roles/monero/tasks/storage.yml | 133 ++++++++++++------- 4 files changed, 116 insertions(+), 63 deletions(-) diff --git a/ansible/host_vars/monerod_box_local/main.yml b/ansible/host_vars/monerod_box_local/main.yml index 5a1b130..c05830b 100644 --- a/ansible/host_vars/monerod_box_local/main.yml +++ b/ansible/host_vars/monerod_box_local/main.yml @@ -12,11 +12,11 @@ monero_restricted_rpc_port: 18089 # The explorer's HTTP port, read by the Caddy play on the edge host. xmrblocks_port: 8081 -# The 500 GB disk added to this VM in Proxmox as scsi1. Named here rather than -# left to the role default so that "which disk holds the blockchain" is a -# property of the machine, recorded next to the machine. The role only ever -# uses it to create the filesystem; the mount itself goes by UUID. -monero_data_device: /dev/sdb +# The 500 GB chain disk, attached in Proxmox as scsi1. Named by that slot, which +# is fixed when the disk is attached and survives reboots and storage moves - +# not /dev/sdb, which it stopped being on 2026-10-04. Only read when the disk has +# no monero-data filesystem yet; see roles/monero/defaults/main.yml. +monero_data_device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 # Systemd services deployed on this host, monitored every 5 minutes. # diff --git a/ansible/roles/monero/README.md b/ansible/roles/monero/README.md index abbe4a1..df9033d 100644 --- a/ansible/roles/monero/README.md +++ b/ansible/roles/monero/README.md @@ -62,10 +62,15 @@ Three things there are load-bearing: - **The whole device, no partition table.** One disk, one job; a partition table is just another layer that can disagree with itself. -- **Everything after mkfs goes by UUID.** `/dev/sdX` ordering is decided by the - kernel at boot. This estate has been bitten eight times by an identifier - written down once whose thing then moved; `monero_data_device` is used *only* - to create the filesystem. +- **The device name is read once, ever.** Each run first looks for a filesystem + labelled `monero-data`; if it exists, `monero_data_device` is not consulted + and the mount goes by that filesystem's UUID. Only a blank disk is found by + name, and that name is the Proxmox slot as published under `/dev/disk/by-id` + (`...drive-scsi1`), fixed before the VM boots and kept across `qm disk move`. + Never `/dev/sdX`: the kernel assigns those in probe order, and on 2026-10-04 a + reboot swapped `sda` and `sdb`, leaving the old `/dev/sdb` value naming the + root disk. The root-disk guard caught it - but a routine run should never + have been reading the name in the first place. - **`RequiresMountsFor=` in the unit.** `nofail` in fstab means a missing disk does not drop a headless VM into an emergency shell - but without `RequiresMountsFor` monerod would then start, find an empty directory, and diff --git a/ansible/roles/monero/defaults/main.yml b/ansible/roles/monero/defaults/main.yml index 3d925ed..0558ce6 100644 --- a/ansible/roles/monero/defaults/main.yml +++ b/ansible/roles/monero/defaults/main.yml @@ -38,16 +38,25 @@ monero_data_dir: /var/lib/monero # The whole device, no partition table: this disk has one job and a partition # table would only add a layer that can disagree with itself. # -# monero_data_device is used ONLY to create the filesystem, once. Everything -# afterwards - fstab, the mount, the systemd unit - goes by UUID, because -# /dev/sdX ordering is decided by the kernel at boot and is not a stable name -# for anything. This estate has been bitten eight times by an identifier that -# was written down once and then moved. -monero_data_device: /dev/sdb +# The disk is known by two names, for two different moments: +# +# monero_data_label is its identity once it holds a filesystem. Every run looks +# for it first, and if it is there the device name is never consulted at all - +# a routine run cannot be broken by a device name, because it does not read one. +# +# monero_data_device is used ONLY on the first run, to find the blank disk and +# create that filesystem. It must be a name fixed before the VM boots: Proxmox +# passes the slot ("scsi1") in as the disk's serial, and udev publishes it under +# /dev/disk/by-id. That survives reboots and `qm disk move` alike. NEVER +# /dev/sdX: the kernel hands those out in probe order, and on 2026-10-04 a reboot +# after moving this disk to the SSD swapped sda and sdb - the value written here +# then named the ROOT disk. +monero_data_label: monero-data +monero_data_device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 monero_data_fstype: ext4 # -m 0: the 5% ext4 reserve exists to stop root-owned daemons deadlocking a full # root filesystem. On a dedicated data disk it is 25 GB thrown away. -monero_data_mkfs_opts: "-m 0 -L monero-data" +monero_data_mkfs_opts: "-m 0 -L {{ monero_data_label }}" # --- Network ---------------------------------------------------------------- # P2P is OUTBOUND ONLY. No inbound port is published from the edge host, so the diff --git a/ansible/roles/monero/tasks/storage.yml b/ansible/roles/monero/tasks/storage.yml index beef59c..e9cb403 100644 --- a/ansible/roles/monero/tasks/storage.yml +++ b/ansible/roles/monero/tasks/storage.yml @@ -5,10 +5,16 @@ # appears nowhere in Ansible, so a rebuild of that host would come up with the # chain silently missing. This role does it in code instead. -- name: Check the Monero data device - ansible.builtin.stat: - path: "{{ monero_data_device }}" - register: monero_device +# Does the disk already carry our filesystem? Its label is the identity from +# here on. When it is found, the device name below is never read: everything +# that follows works from the labelled filesystem, so a stale or reordered +# device name cannot break a routine run. blkid exits 2 for "no such label". +- name: Look for an existing Monero filesystem by its label + ansible.builtin.command: "blkid -L {{ monero_data_label }}" + register: monero_labelled_fs + changed_when: false + failed_when: monero_labelled_fs.rc not in [0, 2] + check_mode: false - name: Find the device backing the root filesystem ansible.builtin.command: findmnt -no SOURCE / @@ -16,48 +22,77 @@ changed_when: false check_mode: false -# A guard, not ceremony. mkfs on the wrong device is unrecoverable, and -# /dev/sdX names are assigned by the kernel at boot - the whole reason -# everything below this point works by UUID. -- name: Assert the data device is a real block device and is not the root disk - ansible.builtin.assert: - that: - - monero_device.stat.exists - - monero_device.stat.isblk | default(false) - - not monero_root_source.stdout.startswith(monero_data_device) - quiet: true - fail_msg: >- - {{ monero_data_device }} is not a usable data device, or it is the disk - the root filesystem lives on ({{ monero_root_source.stdout }}). - Refusing to touch it. Check `lsblk` on the host. +# First run only: the disk is blank and has to be found by name. +- name: Prepare a blank data disk + when: monero_labelled_fs.rc != 0 + block: + # The configured name is a /dev/disk/by-id symlink. Resolve it before + # comparing it to anything: the root source is a kernel name like + # /dev/sdb1, and a symlink path is never a prefix of one, so comparing the + # unresolved name would make the root-disk guard below always pass. + - name: Resolve the configured data device to its kernel name + ansible.builtin.command: "readlink -e {{ monero_data_device }}" + register: monero_device_path + changed_when: false + failed_when: false + check_mode: false -- name: Read any existing filesystem on the data device - ansible.builtin.command: "blkid -o value -s TYPE {{ monero_data_device }}" - register: monero_existing_fs - changed_when: false - failed_when: false - check_mode: false + - name: Check the Monero data device + ansible.builtin.stat: + path: "{{ monero_device_path.stdout | default('', true) or monero_data_device }}" + register: monero_device -# community.general.filesystem is idempotent and will not rebuild a filesystem -# that is already the requested type, but it would happily create one on a disk -# holding something it does not recognise. Say so out loud instead. -- name: Refuse to reformat a disk that already carries a different filesystem - ansible.builtin.assert: - that: - - monero_existing_fs.stdout | trim in ['', monero_data_fstype] - quiet: true - fail_msg: >- - {{ monero_data_device }} already holds a '{{ monero_existing_fs.stdout | trim }}' - filesystem, not '{{ monero_data_fstype }}'. Refusing to overwrite it. + - name: Find the disk the root filesystem lives on + ansible.builtin.command: "lsblk -no PKNAME {{ monero_root_source.stdout }}" + register: monero_root_disk + changed_when: false + check_mode: false -- name: Create the filesystem on the data device - community.general.filesystem: - fstype: "{{ monero_data_fstype }}" - dev: "{{ monero_data_device }}" - opts: "{{ monero_data_mkfs_opts }}" + # A guard, not ceremony. mkfs on the wrong device is unrecoverable. + - name: Assert the data device is a real block device and is not the root disk + ansible.builtin.assert: + that: + - monero_device_path.rc == 0 + - monero_device.stat.exists + - monero_device.stat.isblk | default(false) + - monero_device_path.stdout != '/dev/' ~ (monero_root_disk.stdout | trim) + - not monero_root_source.stdout.startswith(monero_device_path.stdout) + quiet: true + fail_msg: >- + {{ monero_data_device }} (-> {{ monero_device_path.stdout | default('nothing', true) }}) + is not a usable data device, or it is the disk the root filesystem + lives on ({{ monero_root_source.stdout }}). Refusing to touch it. + Check `ls -l /dev/disk/by-id` on the host. + - name: Read any existing filesystem on the data device + ansible.builtin.command: "blkid -o value -s TYPE {{ monero_device_path.stdout }}" + register: monero_existing_fs + changed_when: false + failed_when: false + check_mode: false + + # No monero-data label was found, so a filesystem here is something else. + # community.general.filesystem would leave an ext4 of any label alone but + # happily format over a disk it does not recognise. Say so out loud instead. + - name: Refuse to reformat a disk that already carries a different filesystem + ansible.builtin.assert: + that: + - monero_existing_fs.stdout | trim in ['', monero_data_fstype] + quiet: true + fail_msg: >- + {{ monero_data_device }} already holds a '{{ monero_existing_fs.stdout | trim }}' + filesystem, not '{{ monero_data_fstype }}'. Refusing to overwrite it. + + - name: Create the filesystem on the data device + community.general.filesystem: + fstype: "{{ monero_data_fstype }}" + dev: "{{ monero_device_path.stdout }}" + opts: "{{ monero_data_mkfs_opts }}" + +# By label: on the first run the filesystem was just created with it, on every +# other run it was found by it. - name: Read the filesystem UUID - ansible.builtin.command: "blkid -o value -s UUID {{ monero_data_device }}" + ansible.builtin.command: "blkid -o value -s UUID -t LABEL={{ monero_data_label }}" register: monero_fs_uuid changed_when: false failed_when: false @@ -67,18 +102,22 @@ # without one. On a --check run against a disk that has no filesystem yet this # is expected and unavoidable: check mode did not create the filesystem, so # there is no UUID to read. Say that, rather than leaving a bare blkid rc=2. -- name: Assert the UUID looks like a UUID +# Exactly one: two disks with this label (a cloned disk, say) would make the +# mount below ambiguous. +- name: Assert exactly one UUID was found ansible.builtin.assert: that: - monero_fs_uuid.stdout | trim | length == 36 quiet: true fail_msg: >- - No filesystem UUID on {{ monero_data_device }}. - {% if ansible_check_mode %}This is expected under --check when the disk + Expected exactly one filesystem labelled {{ monero_data_label }}, found: + '{{ monero_fs_uuid.stdout | trim }}'. + {% if ansible_check_mode %}No UUID is expected under --check when the disk has not been formatted yet - check mode cannot create the filesystem, so it cannot dry-run the mount either. Run without --check once, then - --check is meaningful from then on.{% else %}mkfs appears to have failed; - check `lsblk` and `blkid` on the host.{% endif %} + --check is meaningful from then on.{% else %}If none: mkfs appears to have + failed. If several: more than one disk carries the label. Check `lsblk -f` + on the host.{% endif %} # Same owner and mode as the "Own the mounted data directory" task below, and # that is deliberate rather than duplication. Before the disk is mounted this @@ -93,7 +132,7 @@ group: "{{ monero_group }}" mode: '0750' -# By UUID, never by /dev/sdb. nofail so a missing disk does not drop a headless +# By UUID, never by a device name. nofail so a missing disk does not drop a headless # VM into an emergency shell - monerod is stopped from starting by # RequiresMountsFor= in its unit instead, which the systemd monitor then reports. # noatime because LMDB touches a lot of pages and none of it wants an atime write. From f8fe3c6baf082f7e453944ab83784ed5dbac9657 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:22 +0200 Subject: [PATCH 03/15] rebuild: fold monitoring into spacey and prd-arbret into vipy All four droplets were deleted by mistake on 2026-10-17 and rebuilt as two: Gatus and signal-api now share spacey with headscale, and arbret moves onto vipy. The old IPs were recycled to strangers within a day. - inventory: [observability] is spacey. Left as it was, the bare name `monitoring` resolved through ~/.ssh/config to an IP that now belongs to someone else. [arbret]/prd-arbret is gone. - inventory: lan_ip fallbacks corrected from the Proxmox guest agent - memos-box is .130 (was .145) and arbret-staging-box .145 (was .147). - spacey: gatus_port 8081. headscale already holds 0.0.0.0:8080, and the two never shared a host before. - 402: status. is expected to resolve to spacey. - headscale ACL: prd-arbret, prd-monitoring and stalwart-box removed. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/host_vars/monitoring/main.yml | 12 ---------- ansible/host_vars/spacey/main.yml | 7 ++++++ ansible/infra/402_public_monitoring.yml | 2 +- ansible/inventory.ini | 22 +++++++------------ .../services/headscale/templates/acl.json.j2 | 18 +-------------- 5 files changed, 17 insertions(+), 44 deletions(-) delete mode 100644 ansible/host_vars/monitoring/main.yml diff --git a/ansible/host_vars/monitoring/main.yml b/ansible/host_vars/monitoring/main.yml deleted file mode 100644 index 2ef805d..0000000 --- a/ansible/host_vars/monitoring/main.yml +++ /dev/null @@ -1,12 +0,0 @@ ---- - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - caddy diff --git a/ansible/host_vars/spacey/main.yml b/ansible/host_vars/spacey/main.yml index a50d116..b5249e6 100644 --- a/ansible/host_vars/spacey/main.yml +++ b/ansible/host_vars/spacey/main.yml @@ -11,3 +11,10 @@ monitored_services: - headscale - caddy + +# Gatus moved onto this host in the 2026-10-17 rebuild (the `monitoring` droplet +# is gone). Headscale already listens on 0.0.0.0:8080, so Gatus takes 8081. +# Host vars rather than role defaults: the Caddy vhost play in +# services/gatus/deploy_gatus_playbook.yml does not load the role and reads +# this same variable. +gatus_port: 8081 diff --git a/ansible/infra/402_public_monitoring.yml b/ansible/infra/402_public_monitoring.yml index 5aec9c0..a6dfa63 100644 --- a/ansible/infra/402_public_monitoring.yml +++ b/ansible/infra/402_public_monitoring.yml @@ -21,7 +21,7 @@ # then left behind when the machine moved, so the check asserts against # ansible_host - if a box is renumbered, inventory is the one edit. dns_records: - - {sub: "{{ subdomains.gatus }}", host: monitoring} + - {sub: "{{ subdomains.gatus }}", host: spacey} - {sub: "{{ subdomains.headscale }}", host: spacey} - {sub: "{{ subdomains.vaultwarden }}", host: vipy} - {sub: "{{ subdomains.forgejo }}", host: vipy} diff --git a/ansible/inventory.ini b/ansible/inventory.ini index f5964ab..64d0e96 100644 --- a/ansible/inventory.ini +++ b/ansible/inventory.ini @@ -1,7 +1,6 @@ [vps] -vipy ansible_host=167.172.107.33 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -spacey ansible_host=64.227.112.128 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -monitoring ansible_host=64.226.70.190 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua +vipy ansible_host=167.172.98.204 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua +spacey ansible_host=46.101.250.26 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua [nodito_host] nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua @@ -11,9 +10,9 @@ nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ans knots_box_local ansible_host=knots-box lan_ip=192.168.1.135 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua fulcrum_box_local ansible_host=fulcrum-box lan_ip=192.168.1.140 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua mempool_box_local ansible_host=mempool-box lan_ip=192.168.1.142 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -memos_box_local ansible_host=memos-box lan_ip=192.168.1.145 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua +memos_box_local ansible_host=memos-box lan_ip=192.168.1.130 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua forgejo_runner_local ansible_host=forgejo-runner-box lan_ip=192.168.1.132 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.147 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua +arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.145 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua small_backups_local ansible_host=small-backups-box lan_ip=192.168.1.131 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua # lan_ip read off nodito's ARP table for the VM's own MAC (bc:24:11:89:fc:1f), @@ -30,18 +29,13 @@ jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1. [lapy] localhost ansible_connection=local ansible_user=counterweight -[arbret] -prd-arbret ansible_host=167.99.242.62 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua - [edge] vipy -# The group is `observability`, NOT `monitoring` — there is a HOST named -# `monitoring` on line 5, and a group with the same name makes `hosts: monitoring` -# ambiguous. Ansible resolved it to the host and warned: -# [WARNING]: Found both group and host with same name: monitoring +# Gatus and its Signal transport. Shares spacey with headscale since the +# 2026-10-17 rebuild; the separate `monitoring` droplet is gone. [observability] -monitoring +spacey [vpn_control] spacey @@ -78,7 +72,7 @@ forgejo_runner_local localhost # Every machine Ansible may configure as a server. -# Deliberately EXCLUDES [control] (your laptop) and [arbret]. +# Deliberately EXCLUDES [control] (your laptop). [managed:children] vps nodito_host diff --git a/ansible/services/headscale/templates/acl.json.j2 b/ansible/services/headscale/templates/acl.json.j2 index bc60624..2362a1f 100644 --- a/ansible/services/headscale/templates/acl.json.j2 +++ b/ansible/services/headscale/templates/acl.json.j2 @@ -46,12 +46,7 @@ "tag:jar-lid-archive-staging-box:8000", "tag:monerod-box:18089", "tag:monerod-box:8081", - "tag:arbret-staging-box:80", - "tag:stalwart-box:25", - "tag:stalwart-box:465", - "tag:stalwart-box:993", - "tag:stalwart-box:4190", - "tag:stalwart-box:8080" + "tag:arbret-staging-box:80" ], "src": [ "tag:prd-vipy" @@ -80,10 +75,8 @@ { "action": "accept", "dst": [ - "tag:prd-arbret:22", "tag:prd-vipy:22", "tag:memos-box:22", - "tag:stalwart-box:22", "tag:jar-lid-archive-staging-box:22" ], "src": [ @@ -125,20 +118,11 @@ "tag:nodito": [ "{{ headscale_namespace }}@" ], - "tag:prd-arbret": [ - "{{ headscale_namespace }}@" - ], - "tag:prd-monitoring": [ - "{{ headscale_namespace }}@" - ], "tag:prd-vipy": [ "{{ headscale_namespace }}@" ], "tag:small-backups-box": [ "{{ headscale_namespace }}@" - ], - "tag:stalwart-box": [ - "{{ headscale_namespace }}@" ] } } From 9181ff6e10729285c9f4db2f52508a83d4b5a4ad Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:22 +0200 Subject: [PATCH 04/15] ntfy-emergency-app: remove it No longer used. Playbook, vars, the avisame subdomain, its DNS and HTTP checks and the setup docs all go. Co-Authored-By: Claude Opus 5.5 (1M context) --- 02_vps_core_services_setup.md | 16 ----- ansible/group_vars/all/main.yml | 1 - ansible/infra/402_public_monitoring.yml | 2 - .../deploy_ntfy_emergency_app_playbook.yml | 62 ------------------- .../ntfy_emergency_app_vars.yml | 22 ------- ansible/site.yml | 1 - 6 files changed, 104 deletions(-) delete mode 100644 ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml delete mode 100644 ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml diff --git a/02_vps_core_services_setup.md b/02_vps_core_services_setup.md index 5d75de9..76eefd5 100644 --- a/02_vps_core_services_setup.md +++ b/02_vps_core_services_setup.md @@ -285,22 +285,6 @@ LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd n * Start it up again. -## ntfy-emergency-app - -ntfy-emergency-app is a simple web application that allows trusted people to send emergency messages via ntfy notifications. Perfect for situations where you need to be alerted immediately but don't want to enable notifications on your regular messaging apps. - -### Deploy - -* Decide what subdomain you want to serve the emergency app on and update `ansible/services_config.yml` under `ntfy_emergency_app`. - * Note that you will have to add a DNS entry to point to the VPS public IP. -* Configure the ntfy settings in `ntfy_emergency_app_vars.yml`: - * `ntfy_emergency_app_topic`: The ntfy topic to send messages to (default: "emergency") - * `ntfy_emergency_app_ui_message`: Custom message displayed in the web interface -* Ensure `infra_secrets.yml` contains `ntfy_username` and `ntfy_password` with the credentials the app should use. -* Make sure docker is available on the host. -* Run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml`. - - ## Headscale Headscale is a self-hosted Tailscale control server that allows you to create your own Tailscale network. diff --git a/ansible/group_vars/all/main.yml b/ansible/group_vars/all/main.yml index 3a365f8..e9ab735 100644 --- a/ansible/group_vars/all/main.yml +++ b/ansible/group_vars/all/main.yml @@ -36,7 +36,6 @@ subdomains: lnbits: wallet # Secondary services (vipy) - ntfy_emergency_app: avisame personal_blog: pablohere # Memos (memos-box) diff --git a/ansible/infra/402_public_monitoring.yml b/ansible/infra/402_public_monitoring.yml index a6dfa63..ee6eb9f 100644 --- a/ansible/infra/402_public_monitoring.yml +++ b/ansible/infra/402_public_monitoring.yml @@ -26,7 +26,6 @@ - {sub: "{{ subdomains.vaultwarden }}", host: vipy} - {sub: "{{ subdomains.forgejo }}", host: vipy} - {sub: "{{ subdomains.lnbits }}", host: vipy} - - {sub: "{{ subdomains.ntfy_emergency_app }}", host: vipy} - {sub: "{{ subdomains.personal_blog }}", host: vipy} - {sub: "{{ subdomains.memos }}", host: vipy} - {sub: "{{ subdomains.mempool }}", host: vipy} @@ -52,7 +51,6 @@ - {name: vaultwarden, sub: "{{ subdomains.vaultwarden }}", path: "/", status: 200} - {name: forgejo, sub: "{{ subdomains.forgejo }}", path: "/", status: 200} - {name: lnbits, sub: "{{ subdomains.lnbits }}", path: "/", status: 200} - - {name: avisame, sub: "{{ subdomains.ntfy_emergency_app }}", path: "/", status: 200} - {name: blog, sub: "{{ subdomains.personal_blog }}", path: "/", status: 200} - {name: memos, sub: "{{ subdomains.memos }}", path: "/", status: 200} - {name: mempool, sub: "{{ subdomains.mempool }}", path: "/", status: 200} diff --git a/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml b/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml deleted file mode 100644 index 58f0348..0000000 --- a/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml +++ /dev/null @@ -1,62 +0,0 @@ -- name: Deploy ntfy-emergency-app with Docker Compose and configure Caddy reverse proxy - hosts: edge - become: yes - vars_files: - - ./ntfy_emergency_app_vars.yml - vars: - ntfy_emergency_app_subdomain: "{{ subdomains.ntfy_emergency_app }}" - ntfy_emergency_app_domain: "{{ ntfy_emergency_app_subdomain }}.{{ root_domain }}" - # ⚠ UNRESOLVED: this app sends its notifications to an ntfy server, and the - # server it points at ran on watchtower, which is being destroyed. This was - # derived from subdomains.ntfy, which is now gone with it. - # - # Until an ntfy server exists again this URL is dead, and the app fails - # silently at exactly the moment it matters - it is an EMERGENCY notifier. - # Three ways out, none of them automatic: - # * stand ntfy up somewhere else (the monitoring VPS, or vipy) - # * point this at the public ntfy.sh - # * retire the app - ntfy_emergency_app_ntfy_url: "{{ ntfy_service_url }}" - ntfy_emergency_app_ntfy_user: "{{ ntfy_username | default('') }}" - ntfy_emergency_app_ntfy_password: "{{ ntfy_password | default('') }}" - - tasks: - - name: Create ntfy-emergency-app directory - file: - path: "{{ ntfy_emergency_app_dir }}" - state: directory - owner: "{{ ansible_user }}" - group: "{{ ansible_user }}" - mode: '0755' - - - name: Create docker-compose.yml for ntfy-emergency-app - copy: - dest: "{{ ntfy_emergency_app_dir }}/docker-compose.yml" - content: | - version: "3" - services: - ntfy-emergency-app: - image: ghcr.io/pmartincalvo/ntfy-emergency-app:latest - container_name: ntfy-emergency-app - restart: unless-stopped - ports: - - "{{ ntfy_emergency_app_port }}:3000" - environment: - NTFY_TOPIC: "{{ ntfy_emergency_app_topic }}" - NTFY_URL: "{{ ntfy_emergency_app_ntfy_url }}" - NTFY_USER: "{{ ntfy_emergency_app_ntfy_user }}" - NTFY_PASSWORD: "{{ ntfy_emergency_app_ntfy_password }}" - UI_MESSAGE: "{{ ntfy_emergency_app_ui_message }}" - - - name: Deploy ntfy-emergency-app container with docker compose - command: docker compose up -d - args: - chdir: "{{ ntfy_emergency_app_dir }}" - - - name: Publish ntfy-emergency-app through Caddy - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: ntfy-emergency-app - caddy_site_domain: "{{ ntfy_emergency_app_domain }}" - caddy_site_upstream: "localhost:{{ ntfy_emergency_app_port }}" diff --git a/ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml b/ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml deleted file mode 100644 index bfa4bd1..0000000 --- a/ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml +++ /dev/null @@ -1,22 +0,0 @@ -# General -ntfy_emergency_app_dir: /opt/ntfy-emergency-app -ntfy_emergency_app_port: 3000 - -# (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/) - -# ntfy configuration -ntfy_emergency_app_topic: "emergencia" -ntfy_emergency_app_ui_message: "Leave Pablo a message, he will respond as soon as possible" - -# Remote access -remote_host_name: "{{ groups['edge'] | first }}" -remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}" -remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" -remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" -remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}" - -# Where the emergency notifications are sent. This pointed at ntfy.contrapeso.xyz -# on watchtower; that host is being destroyed, so this MUST be repointed before -# the app can work again. Left at the old value so the break is visible rather -# than silently defaulted to something plausible. -ntfy_service_url: "https://ntfy.contrapeso.xyz" diff --git a/ansible/site.yml b/ansible/site.yml index 40f410c..28e620e 100644 --- a/ansible/site.yml +++ b/ansible/site.yml @@ -64,7 +64,6 @@ - import_playbook: services/vaultwarden/deploy_vaultwarden_playbook.yml - import_playbook: services/forgejo/deploy_forgejo_playbook.yml - import_playbook: services/lnbits/deploy_lnbits_playbook.yml -- import_playbook: services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml - import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml # ── Backups: each source dumps itself, the box pulls ──────────────────────── From ffcb0a8672389c43a2ae24560d3d7d65a1ba1385 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:23 +0200 Subject: [PATCH 05/15] headscale: drop the dead Headplane admin route Caddy proxied /admin* to Headplane behind basic auth, but nothing in this repo ever installed Headplane, so the route led nowhere. The vhost is now a plain reverse proxy: the headscale API authenticates every request itself. The headscale_ui_* vault entries are now unused and can be removed. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../headscale/deploy_headscale_playbook.yml | 20 +++---------------- ansible/services/headscale/headscale_vars.yml | 3 --- 2 files changed, 3 insertions(+), 20 deletions(-) diff --git a/ansible/services/headscale/deploy_headscale_playbook.yml b/ansible/services/headscale/deploy_headscale_playbook.yml index fe3a225..29de9b6 100644 --- a/ansible/services/headscale/deploy_headscale_playbook.yml +++ b/ansible/services/headscale/deploy_headscale_playbook.yml @@ -244,23 +244,9 @@ vars: caddy_site_name: headscale caddy_site_domain: "{{ headscale_domain }}" - # Raw body, and it must stay raw: the /admin* route in front of - # Headplane is not expressible as a plain reverse_proxy. The previous - # version of this task wrote only `reverse_proxy localhost:8080`, which - # would have deleted the admin route and its auth on the next run. - caddy_site_body: | - @headplane { - path /admin* - } - handle @headplane { - basicauth { - {{ headscale_ui_username }} {{ headscale_ui_password_hash }} - } - reverse_proxy http://localhost:{{ headplane_port }} - } - # Headscale API is protected by its own API key authentication - # All API operations require a valid Bearer token in the Authorization header - reverse_proxy * http://localhost:{{ headscale_port }} + # The Headscale API authenticates every request itself (Bearer API + # key), so a plain proxy is all it needs. + caddy_site_upstream: "localhost:{{ headscale_port }}" handlers: - name: Restart headscale diff --git a/ansible/services/headscale/headscale_vars.yml b/ansible/services/headscale/headscale_vars.yml index c3fb948..7118cc7 100644 --- a/ansible/services/headscale/headscale_vars.yml +++ b/ansible/services/headscale/headscale_vars.yml @@ -18,6 +18,3 @@ remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_h remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}" - -# Headplane (headscale admin UI), proxied at /admin* behind Caddy basic auth -headplane_port: 3000 From 751277f0c2481370731c8770ca5ff46dee90aa35 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:23 +0200 Subject: [PATCH 06/15] vaultwarden: sign-ups off by default, and publish on loopback only SIGNUPS_ALLOWED was hardcoded 'true' in the compose file and turned off by a separate manual playbook, so every routine run of the deploy - site.yml included - silently re-opened registration. It is now vaultwarden_signups_allowed, default false. The port was published as a bare "8222:80". Docker writes its own iptables rules ahead of UFW, so Vaultwarden answered plain HTTP on the public interface, bypassing Caddy and TLS. Now 127.0.0.1:8222; verified closed from outside. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../services/vaultwarden/deploy_vaultwarden_playbook.yml | 7 +++++-- ansible/services/vaultwarden/vaultwarden_vars.yml | 7 +++++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml b/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml index 282fc2e..0a4a080 100644 --- a/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml +++ b/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml @@ -27,13 +27,16 @@ container_name: vaultwarden restart: unless-stopped ports: - - "{{ vaultwarden_port }}:80" + # Loopback only. Docker writes its own iptables rules AHEAD of + # UFW, so a bare "8222:80" was reachable from the internet as + # plain HTTP, bypassing Caddy and TLS. Caddy is the only client. + - "127.0.0.1:{{ vaultwarden_port }}:80" volumes: - ./data:/data environment: WEBSOCKET_ENABLED: 'true' DOMAIN: "https://{{ vaultwarden_domain }}" - SIGNUPS_ALLOWED: 'true' + SIGNUPS_ALLOWED: '{{ vaultwarden_signups_allowed | bool | lower }}' LOG_FILE: /data/vaultwarden.log - name: Deploy vaultwarden container with docker compose diff --git a/ansible/services/vaultwarden/vaultwarden_vars.yml b/ansible/services/vaultwarden/vaultwarden_vars.yml index 9edc2cd..c7470f5 100644 --- a/ansible/services/vaultwarden/vaultwarden_vars.yml +++ b/ansible/services/vaultwarden/vaultwarden_vars.yml @@ -3,6 +3,13 @@ vaultwarden_dir: /opt/vaultwarden vaultwarden_data_dir: "{{ vaultwarden_dir }}/data" vaultwarden_port: 8222 +# Public registration. OFF: the accounts exist, and this vault is on the open +# internet. It used to be hardcoded 'true' in the compose file and turned off by +# a separate manual playbook - so every routine run of the deploy (site.yml +# included) silently re-opened sign-ups. Flip to true only to add an account, +# then back. +vaultwarden_signups_allowed: false + # (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/) # Remote access From 532974634e436e9d16820dec42f9b97c57151f5c Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:23 +0200 Subject: [PATCH 07/15] playbooks: stop re-runs overwriting live config Three seeding tasks used copy without force: false, so every run replaced the real file with the starter one: - personal-blog: the placeholder index.html over the deployed site. - lnbits: .env.example over .env - a new AUTH_SECRET_KEY, logging every user out, and it would have clobbered a restored .env. - forgejo: a two-line app.ini stub over the real config, losing INTERNAL_TOKEN, the JWT secrets, the DB path and ROOT_URL. All three now seed a fresh install only. Found while restoring each from backup on 2026-10-17. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/services/forgejo/deploy_forgejo_playbook.yml | 4 ++++ ansible/services/lnbits/deploy_lnbits_playbook.yml | 4 ++++ .../services/personal-blog/deploy_personal_blog_playbook.yml | 3 +++ 3 files changed, 11 insertions(+) diff --git a/ansible/services/forgejo/deploy_forgejo_playbook.yml b/ansible/services/forgejo/deploy_forgejo_playbook.yml index f908586..29009ca 100644 --- a/ansible/services/forgejo/deploy_forgejo_playbook.yml +++ b/ansible/services/forgejo/deploy_forgejo_playbook.yml @@ -66,6 +66,10 @@ owner: "{{ forgejo_user }}" group: "{{ forgejo_user }}" mode: '0644' + # Fresh install only: the web installer then writes the real config + # (INTERNAL_TOKEN, JWT secrets, DB path, ROOT_URL) into this file. + # Without force: false every re-run replaced it with this stub. + force: false - name: Download Forgejo systemd service file get_url: diff --git a/ansible/services/lnbits/deploy_lnbits_playbook.yml b/ansible/services/lnbits/deploy_lnbits_playbook.yml index 7b89f33..d0e1a77 100644 --- a/ansible/services/lnbits/deploy_lnbits_playbook.yml +++ b/ansible/services/lnbits/deploy_lnbits_playbook.yml @@ -85,6 +85,10 @@ owner: "{{ ansible_user }}" group: "{{ ansible_user }}" mode: '0644' + # Seed a fresh install only. Without this every re-run replaced the real + # .env (AUTH_SECRET_KEY and all) with the example, logging every user + # out - and would have clobbered a restored one. + force: false - name: Configure LNBits environment variables diff --git a/ansible/services/personal-blog/deploy_personal_blog_playbook.yml b/ansible/services/personal-blog/deploy_personal_blog_playbook.yml index 21d8b43..d3a9951 100644 --- a/ansible/services/personal-blog/deploy_personal_blog_playbook.yml +++ b/ansible/services/personal-blog/deploy_personal_blog_playbook.yml @@ -45,6 +45,9 @@ owner: "{{ ansible_user }}" group: www-data mode: '0664' + # Only on a fresh web root. Without this, every re-run (and site.yml) + # overwrote the real deployed index.html with the placeholder. + force: false - name: Publish the blog through Caddy ansible.builtin.include_role: From f33f1a4b67cc167e13012d3fa12cf46c4f69dc54 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:24 +0200 Subject: [PATCH 08/15] monero: check every five minutes again The probes, the unit check and the public checks ran once a day with 26h windows, justified as "a multi-week initial sync on a rotating disk". That stopped being true: the node is synced, and a cold explorer page now renders in 29s and a warm one in 0.2s - polling every 5 minutes is what keeps it warm. A dead monerod could go unnoticed for a day. Back to the standard 5m cadence and 11m heartbeat; the per-host service_health override on monerod-box is gone. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/host_vars/monerod_box_local/main.yml | 21 ------------------- ansible/infra/401_service_monitoring.yml | 6 +++--- ansible/infra/402_public_monitoring.yml | 19 ++++------------- .../infra/403_service_probe_registration.yml | 15 ++++++------- ansible/roles/monero/defaults/main.yml | 10 +++++---- .../monero/templates/healthcheck.timer.j2 | 16 ++++++-------- ansible/roles/xmrblocks/defaults/main.yml | 11 +++++----- .../xmrblocks/templates/healthcheck.timer.j2 | 11 +++++----- 8 files changed, 36 insertions(+), 73 deletions(-) diff --git a/ansible/host_vars/monerod_box_local/main.yml b/ansible/host_vars/monerod_box_local/main.yml index c05830b..d9bfef4 100644 --- a/ansible/host_vars/monerod_box_local/main.yml +++ b/ansible/host_vars/monerod_box_local/main.yml @@ -29,24 +29,3 @@ monero_data_device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 monitored_services: - monerod - xmrblocks - -# ...but on THIS host, once a day rather than every five minutes. -# -# Both units are a Monero node in the middle of a multi-week initial sync on a -# rotating disk. There is nothing a 5-minute poll reports that a daily one does -# not, and every failing poll was another Signal message - 83 of them in six -# hours while the explorer was wedged. -# -# This is safe to scope to the host because monerod-box runs ONLY Monero units. -# Do not copy it to a box with a mixed workload: the timer is one per host and -# iterates every unit in monitored_services, so slowing it slows all of them. -# -# 26h, not 24h: the window must EXCEED the push interval, or a check reporting -# exactly on schedule races its own deadline and alerts on jitter alone. The -# 900s randomized delay puts the real worst case at ~24h15m. -# -# Host liveness and disk are deliberately NOT slowed - they answer "is the box -# alive", which is still wanted within minutes. See infra/400. -service_health_on_calendar: "*-*-* 05:15:00" -service_health_randomized_delay: "900" -service_health_heartbeat: "26h" diff --git a/ansible/infra/401_service_monitoring.yml b/ansible/infra/401_service_monitoring.yml index 94828eb..c181ba1 100644 --- a/ansible/infra/401_service_monitoring.yml +++ b/ansible/infra/401_service_monitoring.yml @@ -99,9 +99,9 @@ healthcheck_units: "{{ monitored_services }}" healthcheck_units_key_prefix: "services_{{ host_key }}" # Cadence and heartbeat window are ONE decision, and both are a property - # of the host - monerod-box runs a multi-week sync nobody needs polled - # every five minutes, so it declares a daily schedule in its host_vars - # and a matching window above. Everything else keeps 5min/11m. + # of the host: a host can set service_health_on_calendar and a matching + # service_health_heartbeat in its host_vars. None does today; everything + # runs 5min/11m. (monerod-box was daily during its initial sync.) # # An empty healthcheck_on_calendar means "use healthcheck_interval" - # see the timer template in roles/healthcheck. diff --git a/ansible/infra/402_public_monitoring.yml b/ansible/infra/402_public_monitoring.yml index ee6eb9f..d572171 100644 --- a/ansible/infra/402_public_monitoring.yml +++ b/ansible/infra/402_public_monitoring.yml @@ -30,11 +30,8 @@ - {sub: "{{ subdomains.memos }}", host: vipy} - {sub: "{{ subdomains.mempool }}", host: vipy} - {sub: "{{ subdomains.datum_gateway }}", host: vipy} - # Daily rather than the 6h default - see the Monero note under - # public_sites. A record that has not moved in a year does not need - # four looks a day while the node behind it is mid-sync. - - {sub: "{{ subdomains.monero }}", host: vipy, interval: '24h'} - - {sub: "{{ subdomains.monero_explorer }}", host: vipy, interval: '24h'} + - {sub: "{{ subdomains.monero }}", host: vipy} + - {sub: "{{ subdomains.monero_explorer }}", host: vipy} - {sub: "{{ subdomains.music }}", host: vipy} - {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy} @@ -58,16 +55,8 @@ # 401 is CORRECT: monerod answers an unauthenticated request with an HTTP # Digest challenge. Asserting 200 here would go green precisely when # rpc-login broke and the node started serving the world for free. - # - # Both Monero sites are polled ONCE A DAY, not every 5 minutes. The node - # is a multi-week initial sync on a rotating disk and the explorer reads - # the same LMDB, so a hard poll measures the disk more than the service - - # and a wedged explorer turned that cadence into 83 Signal messages in - # six hours. `threshold` drops with it: 3 failures at 5m is 15 minutes of - # tolerance, but at 24h it would be THREE DAYS before anyone heard. A - # daily check has no room for a blip, so it alerts on the first one. - - {name: monero, sub: "{{ subdomains.monero }}", path: "/get_info", status: 401, interval: '24h', threshold: 1} - - {name: exmrplore, sub: "{{ subdomains.monero_explorer }}", path: "/", status: 200, interval: '24h', threshold: 1} + - {name: monero, sub: "{{ subdomains.monero }}", path: "/get_info", status: 401} + - {name: exmrplore, sub: "{{ subdomains.monero_explorer }}", path: "/", status: 200} # /app/, not "/": checked live, "/" answers 302 (Navidrome redirects to # the SPA) and only /app/ returns 200 with the real UI. 200 rather than # 401 because Navidrome has no basic auth in front of it - the Subsonic diff --git a/ansible/infra/403_service_probe_registration.yml b/ansible/infra/403_service_probe_registration.yml index e1ed107..b716faa 100644 --- a/ansible/infra/403_service_probe_registration.yml +++ b/ansible/infra/403_service_probe_registration.yml @@ -45,15 +45,12 @@ - {name: mempool-mariadb, host: mempool_box_local} - {name: mempool-backend, host: mempool_box_local} - {name: mempool-frontend, host: mempool_box_local} - # The two Monero probes push ONCE A DAY, not every 5 minutes, so they - # need a window wider than a day. 26h, not 24h: the window must exceed - # the push interval or a check that reports exactly on time races its own - # deadline and alerts on jitter. The timers are OnCalendar 05:00 and - # 05:30 with RandomizedDelaySec=900, so the true worst case is ~24h15m. - # Cadence lives in roles/{monero,xmrblocks}/defaults/main.yml; the two - # are one decision and must move together. - - {name: monerod, host: monerod_box_local, heartbeat: '26h'} - - {name: xmrblocks, host: monerod_box_local, heartbeat: '26h'} + # Every 5 minutes like the rest since 2026-10-17 (they were daily while + # the node synced on a rotating disk). Cadence lives in + # roles/{monero,xmrblocks}/defaults/main.yml; it and this window are one + # decision and must move together. + - {name: monerod, host: monerod_box_local} + - {name: xmrblocks, host: monerod_box_local} - {name: navidrome, host: music_box_local} - {name: jar-lid-archive-staging, host: jar_lid_staging_local} diff --git a/ansible/roles/monero/defaults/main.yml b/ansible/roles/monero/defaults/main.yml index 0558ce6..686b1c9 100644 --- a/ansible/roles/monero/defaults/main.yml +++ b/ansible/roles/monero/defaults/main.yml @@ -104,15 +104,17 @@ healthcheck_push_url: "" # is set: Gatus rejects an unauthenticated push with 401. healthcheck_push_token: "" -# WHEN TO CHECK. Daily, at a fixed hour, rather than an OnUnitActiveSec -# interval that drifts with every restart. +# WHEN TO CHECK. Every 5 minutes, on the calendar rather than OnUnitActiveSec so +# the cadence does not drift with every restart. (It was daily while the node +# did its initial sync on a rotating disk; since 2026-10-03 the chain is on NVMe +# and fully synced, and get_info answers in well under a second.) # # Prefixed because the generic `healthcheck` role owns the unprefixed # healthcheck_on_calendar; an unprefixed default here would shadow it for any # play that includes both. # -# Paired with the 26h Gatus heartbeat in +# Paired with the standard 11m Gatus heartbeat in # infra/403_service_probe_registration.yml. Change one and you must change the # other: the window has to stay above the interval or the check alerts on its # own jitter. -monero_healthcheck_on_calendar: "*-*-* 05:00:00" +monero_healthcheck_on_calendar: "*:0/5" diff --git a/ansible/roles/monero/templates/healthcheck.timer.j2 b/ansible/roles/monero/templates/healthcheck.timer.j2 index c088af5..306a575 100644 --- a/ansible/roles/monero/templates/healthcheck.timer.j2 +++ b/ansible/roles/monero/templates/healthcheck.timer.j2 @@ -8,17 +8,13 @@ Description=Monero Node Health Check Timer [Timer] OnBootSec=2min -# ONCE A DAY, not every 5 minutes. -# -# A 5-minute poll of monerod buys nothing a daily one does not: the node is a -# multi-week initial sync on a rotating disk, and every failing poll is another -# Signal message. The cadence and the Gatus heartbeat window are ONE decision - -# the window MUST exceed this interval, and it is set to 26h in -# infra/403_service_probe_registration.yml. A 24h push under a 24h window races -# itself and alerts on jitter alone. +# Every 5 minutes (see monero_healthcheck_on_calendar). The cadence and the +# Gatus heartbeat window are ONE decision - the window MUST exceed this +# interval, and it is the standard 11m in +# infra/403_service_probe_registration.yml. OnCalendar={{ monero_healthcheck_on_calendar }} -# Spread against the other daily pushes leaving this box. -RandomizedDelaySec=900 +# Small: a delay comparable to the interval would eat the heartbeat's slack. +RandomizedDelaySec=30 Persistent=true [Install] diff --git a/ansible/roles/xmrblocks/defaults/main.yml b/ansible/roles/xmrblocks/defaults/main.yml index 8254393..8661676 100644 --- a/ansible/roles/xmrblocks/defaults/main.yml +++ b/ansible/roles/xmrblocks/defaults/main.yml @@ -74,8 +74,9 @@ healthcheck_push_url: "" # is set: Gatus rejects an unauthenticated push with 401. healthcheck_push_token: "" -# WHEN TO CHECK. Daily, 30 minutes after monerod's own check so the two do not -# contend for the same cold LMDB pages. See roles/monero/defaults for why the -# name is prefixed and why this is paired with the 26h heartbeat in -# infra/403_service_probe_registration.yml. -xmrblocks_healthcheck_on_calendar: "*-*-* 05:30:00" +# WHEN TO CHECK. Every 5 minutes, offset 2 minutes from monerod's own check so +# the two do not land together. On NVMe a cold page render measured 29s and a +# warm one 0.2s (2026-10-17); a 5-minute poll is what keeps it warm. See +# roles/monero/defaults for why the name is prefixed and why this is paired +# with the 11m heartbeat in infra/403_service_probe_registration.yml. +xmrblocks_healthcheck_on_calendar: "*:2/5" diff --git a/ansible/roles/xmrblocks/templates/healthcheck.timer.j2 b/ansible/roles/xmrblocks/templates/healthcheck.timer.j2 index 6edf61d..b6c9ba8 100644 --- a/ansible/roles/xmrblocks/templates/healthcheck.timer.j2 +++ b/ansible/roles/xmrblocks/templates/healthcheck.timer.j2 @@ -7,14 +7,13 @@ Description=Onion Monero Blockchain Explorer Health Check Timer [Timer] OnBootSec=2min -# ONCE A DAY, not every 5 minutes. See the note in roles/monero's timer: the -# explorer follows the node it reads from, and this check costs a full page -# render out of the LMDB (a cold one measured 68 seconds), so polling it hard -# is not free either. The Gatus heartbeat window is 26h, set in +# Every 5 minutes (see xmrblocks_healthcheck_on_calendar). The check renders a +# full page out of the LMDB: 29s cold, 0.2s warm on NVMe - frequent polling is +# what keeps it warm. The Gatus heartbeat window is the standard 11m, set in # infra/403_service_probe_registration.yml, and MUST stay above this interval. OnCalendar={{ xmrblocks_healthcheck_on_calendar }} -# Spread against the other daily pushes leaving this box. -RandomizedDelaySec=900 +# Small: a delay comparable to the interval would eat the heartbeat's slack. +RandomizedDelaySec=30 Persistent=true [Install] From a3d3f063f8d4cf492e9b6e45fb983c8f1d4a8c23 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:24 +0200 Subject: [PATCH 09/15] gatus: service checks tolerate one failed run The systemd unit checks alerted on failure-threshold 1, on the reasoning that the heartbeat window is the tolerance. That covers a MISSING push, but a unit caught mid-restart - a deploy, an upgrade, a backup stopping it - pushes an explicit false, and each of those paged and resolved ten minutes later. 2 means down on two consecutive 5-minute runs. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/infra/401_service_monitoring.yml | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/ansible/infra/401_service_monitoring.yml b/ansible/infra/401_service_monitoring.yml index c181ba1..088cc7b 100644 --- a/ansible/infra/401_service_monitoring.yml +++ b/ansible/infra/401_service_monitoring.yml @@ -44,6 +44,13 @@ # # Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints # have no built-in tolerance, so the threshold is where it belongs for them. +# +# EXCEPT here, where the threshold is 2. The heartbeat only forgives a MISSING +# push. These checks also push an explicit `false` whenever a unit is caught +# mid-restart - a deploy, an unattended upgrade, a backup stopping the service +# for its dump - and at 1 every one of those pages, then resolves ten minutes +# later. 2 means "down on two consecutive 5-minute runs", i.e. 5-10 minutes. +# A host that stops reporting altogether still alarms on the heartbeat alone. # ───────────────────────────────────────────────────────────────────────────── - name: Register the service checks with Gatus hosts: observability @@ -73,8 +80,9 @@ vars: gatus_endpoint_default_alerts: - type: signal - # 1, not 3: the heartbeat window is the tolerance. See the note above. - failure-threshold: 1 + # 2: tolerate one explicit `false` from a unit caught mid-restart. + # See the note above. + failure-threshold: 2 success-threshold: 2 send-on-resolved: true minimum-reminder-interval: 6h From fba533ebb1a67ad35721cf69477755e580688f6a Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:24 +0200 Subject: [PATCH 10/15] signal-api: bump to 0.101 - Signal refused the old client Relinking after the rebuild failed: the phone said "network error" and signal-cli logged "Link request error: Connection closed!", with the network itself fine (TLS to chat.signal.org completed). The pinned 0.100 bundled signal-cli 0.14.5 from June; Signal stops accepting clients that fall a few months behind. 0.101 (signal-cli 0.14.8) linked first time. A stale pin fails the same way for alerts - silently - so the pin now carries signal_api_version beside the digest and a note to bump it every couple of months. The old comment saying upstream publishes no versioned tags was wrong. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/roles/signal_api/defaults/main.yml | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/ansible/roles/signal_api/defaults/main.yml b/ansible/roles/signal_api/defaults/main.yml index 6bb7d4f..5b1b79f 100644 --- a/ansible/roles/signal_api/defaults/main.yml +++ b/ansible/roles/signal_api/defaults/main.yml @@ -4,12 +4,19 @@ # Gatus does not speak Signal. It POSTs JSON to this service, which holds the # actual Signal identity and does the protocol work. -# Pinned by digest for the same reason as Gatus: a tag is mutable. -# Upstream publishes no versioned tags worth pinning to, so this pins the -# DIGEST that `latest` resolved to when this was reviewed. `latest` is a moving -# target; a digest is a content address, and `docker compose pull` either -# fetches exactly this image or fails. -signal_api_image_digest: "sha256:2399d449123cdad56c4d859277e3b9127e1a00c4d2ab4601c239882609286cf8" +# Pinned by DIGEST, with the release tag alongside for humans - same as Gatus. +# A tag is mutable; a digest is a content address, so `docker compose pull` +# either fetches exactly this image or fails. Upstream publishes versioned +# release tags (0.97 ... 0.101); the two values below must move together. +# +# THIS PIN EXPIRES. Signal's servers stop accepting signal-cli releases that +# fall a few months behind. 0.100 (signal-cli 0.14.5, 2026-06-11) was refused +# on 2026-10-18 - the phone reported "network error" and signal-cli logged +# "Link request error: Connection closed!", with the network itself fine. A +# stale pin fails exactly the same way for ALERTS: silently. Bump this at least +# every couple of months: https://github.com/bbernhard/signal-cli-rest-api/releases +signal_api_version: "0.101" # bundles signal-cli 0.14.8 +signal_api_image_digest: "sha256:2cf09d66a86fdd210e42ee91662216364acc27da1163a8c44167a77ea3a6559a" signal_api_image: "bbernhard/signal-cli-rest-api@{{ signal_api_image_digest }}" signal_api_dir: /opt/signal-api From d19553f587cb105b7c7f2d3d83c59c876ec59ebe Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:25 +0200 Subject: [PATCH 11/15] gatus: turn Signal alerting back on signal-api is relinked (test message delivered; the vault's signal_number matches the linked account). No endpoint overrides `enabled`, so this one key still switches every alert in the estate. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/services/gatus/deploy_gatus_playbook.yml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/ansible/services/gatus/deploy_gatus_playbook.yml b/ansible/services/gatus/deploy_gatus_playbook.yml index 94cd325..26b8c8e 100644 --- a/ansible/services/gatus/deploy_gatus_playbook.yml +++ b/ansible/services/gatus/deploy_gatus_playbook.yml @@ -22,9 +22,10 @@ number: "{{ signal_number }}" recipients: "{{ signal_recipients }}" default-alert: - # ── SIGNAL ALERTING IS OFF ────────────────────────────────────── - # Flip to true to turn the phone back on. Nothing else needs to - # change: this ONE key disables every Signal alert in the estate. + # ── SIGNAL ALERTING IS ON ─────────────────────────────────────── + # Flip to false to silence the phone. Nothing else needs to + # change: this ONE key switches every Signal alert in the estate. + # (Re-enabled 2026-10-18 after relinking signal-api post-rebuild.) # # It works because `enabled` is the only alert field no registration # playbook sets. Gatus merges a provider's default-alert into each @@ -36,7 +37,7 @@ # Deliberately NOT done by deleting the `signal:` provider block: # that would strand ~90 endpoints referencing a provider that no # longer exists, and lose the number and recipients. - enabled: false + enabled: true # Overridden per group by the registration playbooks; these are the # values that apply if a caller sets nothing. failure-threshold: 1 From 2a80f0e3cad5815073842ee5672251516e73c0ef Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:25 +0200 Subject: [PATCH 12/15] bitcoin-knots: ride out block-time RPC stalls The RPC probe flapped while every other knots check stayed green. Each burst lined up with a block: validation on the saturated HDD holds the node busy while Fulcrum, mempool and DATUM all fetch the block, and bitcoind rejected calls with "work queue depth exceeded" (93 times on 2026-10-18). - 403: the probe runs every minute, so failure-threshold 4 rides out a 1-3 minute burst while a real outage still alerts within ~5 minutes. Probes now take an optional failure_threshold over a shared probe_alert. - rpcworkqueue=64 and rpcthreads=8, so bursts queue instead of failing. Profiling later showed the stalls are per-input UTXO lookups at ~5 ms each - HDD seek latency - so this treats the symptom. The fix is chainstate on SSD. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../infra/403_service_probe_registration.yml | 28 +++++++++++++------ ansible/roles/bitcoin_knots/defaults/main.yml | 9 ++++++ .../bitcoin_knots/templates/bitcoin.conf.j2 | 2 ++ 3 files changed, 30 insertions(+), 9 deletions(-) diff --git a/ansible/infra/403_service_probe_registration.yml b/ansible/infra/403_service_probe_registration.yml index b716faa..a46682a 100644 --- a/ansible/infra/403_service_probe_registration.yml +++ b/ansible/infra/403_service_probe_registration.yml @@ -37,7 +37,13 @@ vars: probes: - - {name: bitcoin-knots, host: knots_box_local} + # failure_threshold 4: this probe runs every MINUTE, and bitcoind turns RPC + # away for 1-3 minutes when a block lands - validation on the saturated + # HDD plus Fulcrum/mempool/DATUM all fetching the block at once (93 + # "work queue depth exceeded" in one day, 2026-10-18). At 1, every slow + # block paged. 4 consecutive failed minutes is past any block burst seen, + # and a real outage still alerts within ~5 minutes. + - {name: bitcoin-knots, host: knots_box_local, failure_threshold: 4} - {name: datum-gateway, host: knots_box_local} - {name: fulcrum, host: fulcrum_box_local} - {name: phoenixd, host: vipy} @@ -54,6 +60,16 @@ - {name: navidrome, host: music_box_local} - {name: jar-lid-archive-staging, host: jar_lid_staging_local} + # The alert every probe gets. failure-threshold is 1 - the heartbeat window + # is the tolerance, see the note above - unless a probe sets + # failure_threshold, for checks that legitimately fail for a minute or two. + probe_alert: + type: signal + failure-threshold: 1 + success-threshold: 2 + send-on-resolved: true + minimum-reminder-interval: 6h + tasks: - name: Build the probe endpoint list ansible.builtin.set_fact: @@ -61,19 +77,13 @@ 'name': item.name, 'group': 'probe', 'token': gatus_push_tokens[item.host], - 'heartbeat': item.heartbeat | default('11m')}] }}" + 'heartbeat': item.heartbeat | default('11m'), + 'alerts': [probe_alert | combine({'failure-threshold': item.failure_threshold | default(1)})]}] }}" loop: "{{ probes }}" - name: Register the probe endpoints ansible.builtin.include_role: name: gatus_endpoint vars: - gatus_endpoint_default_alerts: - - type: signal - # 1, not 3: the heartbeat window is the tolerance. See the note above. - failure-threshold: 1 - success-threshold: 2 - send-on-resolved: true - minimum-reminder-interval: 6h gatus_endpoint_name: probes gatus_endpoint_external: "{{ probe_endpoints }}" diff --git a/ansible/roles/bitcoin_knots/defaults/main.yml b/ansible/roles/bitcoin_knots/defaults/main.yml index 1045e77..01a352d 100644 --- a/ansible/roles/bitcoin_knots/defaults/main.yml +++ b/ansible/roles/bitcoin_knots/defaults/main.yml @@ -22,6 +22,15 @@ bitcoin_rpc_port: 8332 bitcoin_p2p_port: 8333 bitcoin_rpc_bind: "0.0.0.0" +# RPC capacity. When a block lands, Fulcrum, mempool and DATUM all fetch it at +# once while validation holds the node busy on a saturated HDD; with the +# defaults the queue overflowed and bitcoind REJECTED calls ("http work queue +# depth exceeded" - 93 times on 2026-10-18), which also made the health probe +# flap. A deeper queue lets those bursts wait instead of failing. Costs a few +# idle threads and a little memory. +bitcoin_rpcworkqueue: 64 +bitcoin_rpcthreads: 8 + # Build options bitcoin_build_jobs: 4 # Parallel build jobs (-j flag), adjust based on CPU cores bitcoin_build_prefix: /usr/local diff --git a/ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2 b/ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2 index 5277a3c..4187e24 100644 --- a/ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2 +++ b/ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2 @@ -11,6 +11,8 @@ rpcpassword={{ bitcoin_rpc_password }} rpcbind={{ bitcoin_rpc_bind }} rpcport={{ bitcoin_rpc_port }} rpcallowip=0.0.0.0/0 +rpcworkqueue={{ bitcoin_rpcworkqueue }} +rpcthreads={{ bitcoin_rpcthreads }} # Network Configuration listen=1 From d50774c39a39900bf0a1b9c595ee99d6cbb9c7c1 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:25 +0200 Subject: [PATCH 13/15] bitcoin-knots: raise dbcache from 200 to 1200 Tried on 2026-10-18 to cut the HDD reads: at 200 MB the P2P thread alone read ~4 MB/s validating relayed transactions. Steady-state reads did drop (to ~0.4 MB/s) and RPC rejections went to zero. But block validation got SLOWER over the next 14 blocks: median 60s against 11s before. bitcoind's bench log puts all of it in input lookups at ~5 ms each. The likely cause is that the extra GB came out of the guest's page cache, which holds compressed chainstate files far more densely than bitcoind's cache holds decoded coins. Committed because it is what is running; probably to be reverted. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/roles/bitcoin_knots/defaults/main.yml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/ansible/roles/bitcoin_knots/defaults/main.yml b/ansible/roles/bitcoin_knots/defaults/main.yml index 01a352d..cce3850 100644 --- a/ansible/roles/bitcoin_knots/defaults/main.yml +++ b/ansible/roles/bitcoin_knots/defaults/main.yml @@ -69,11 +69,17 @@ healthcheck_push_token: "" bitcoin_logfile: "{{ bitcoin_data_dir }}/debug.log" # --- dbcache ---------------------------------------------------------------- -# Computed as 90% of RAM unless this is set. The live node was hand-tuned to -# 200 MB; the calculation would have produced 3528. As with fulcrum, note that +# Computed as 90% of RAM unless this is set. As with fulcrum, note that # set_fact outranks role defaults, so the CALCULATION has to honour this - a # value pinned only in defaults/ is silently ignored. -bitcoin_dbcache_mb_override: 200 +# +# 1200, raised from a hand-tuned 200 on 2026-10-18. At 200 MB almost no UTXO +# lookup hit memory: the P2P thread alone read ~4 MB/s off the saturated HDD +# mirror validating relayed transactions, and block validation took a median +# 11 s, p90 103 s, max 344 s - long enough for RPC to stall and the health +# probe to fail. 1200 is what fits in the 4 GB VM (bitcoind ~1.5 GB RSS at +# 200, mempool ~3 MB in practice). Raising the VM to 6 GB would allow ~3000. +bitcoin_dbcache_mb_override: 1200 # --- DATUM Gateway ---------------------------------------------------------- # This node feeds block templates to datum_gateway on knots-box. These settings From 3be5e7934c94d02e466e9ae3f5762766785b0302 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:26 +0200 Subject: [PATCH 14/15] gatus: watch the 99stack runway 99stack is prepaid: when the balance runs out the servers stop. A pulled check on the account API alerts when billing.secondsdue - the provider's own runway at current spend - drops under 14 days: checked every 6h, reminding every 24h. gatus_endpoint gains a `headers` field for APIs behind a credential. The vault gains ninetyninestack_api_token (account-wide; 99stack has no read-only scope) and gatus_push_tokens.arbret, used by the next commit. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/group_vars/all/vault.yml | 445 +++++++++--------- ansible/infra/402_public_monitoring.yml | 46 +- .../roles/gatus_endpoint/defaults/main.yml | 1 + .../templates/endpoints.yaml.j2 | 4 + 4 files changed, 277 insertions(+), 219 deletions(-) diff --git a/ansible/group_vars/all/vault.yml b/ansible/group_vars/all/vault.yml index dd06bdc..a3bdbd7 100644 --- a/ansible/group_vars/all/vault.yml +++ b/ansible/group_vars/all/vault.yml @@ -1,218 +1,229 @@ $ANSIBLE_VAULT;1.1;AES256 -33653731643464333331646636613630383836323134363761333038383334353230663631376666 -3861376232363238613234663263646137336662666263360a633661313435616338383337396664 -31316234616365326363353739323230326264623562313061303339373265663435313938326634 -3464636635386664620a663330396463653963613537653230333430326137333964386530343636 -62383336343031363537383362643934643839653731346364636534373430303636353431646534 -66313264343338623333633964376137623833613664393137343432666264656466353830346631 -62616639343265646661373062376138363232376562316134303730323431336533306565646438 -62666132613930646564393162316461643431666664356436356264633839333338356561333038 -65333466396165363030653137663939303838663030666235356639616362346330346437306133 -61653130363136643038626562653437303939393133613733326530656163323531386436656363 -37303930613063366537623435396564643431346266373363333332663632663939346131373565 -32663430643034663530643431346335623636663265316431336361363233383761333130353530 -30373566316336646536313938353663666436643532303433643139376232646566616430366330 -36646433393037373733633562306631623164356237326532653833633031633735653463343461 -65646664383531346531353133313334663130393237353436623638333731326139653363383032 -63356235646366333534366666383961636361626636616237323237346234383035623365376137 -34646162663761343662353631393131313562326138336565383764666333383731353730376135 -64656231613630646330643133373766643330656166346134363263373966363665323964396233 -64333038373565343330626633323933333239623634333436666338633539316263616438626461 -30643566656534666533373365613861383231373062613831653537623532353561633736633866 -61383532633430323330646137613633383830393136303332643531383466353039353564393232 -39326130333336663038643766303464363664333261313137663863616466666534383831613732 -35303831633232306131636137656461653361346134373935333462306232323165313265323239 -36343864306130303537386164653866656632633861363736336362656161356463386265336430 -38323739336364303061363437346137633436653566353464323462643838313631393235643635 -65626461643263656537616638623066346636663335383861376336623932343439353231343739 -64663263613862626266616362313136373630636262633036386462666632623336343431333231 -31333164356538616562353335313161353339653262616439636134383063346566663261323938 -36633631636265396230313033373230373635643165613239623532613436376464646635326535 -39333433336361343530623434356636363433653035333934653162363135626564346161346130 -65313130393639393539633534653330343935376365656162383132643635313034383538333333 -64346336346231313939396463643166363165393439626431386334393563646337383361326431 -34346462333732643365303230383937316462623566346238393431656433323535376361623561 -66333530346638623332633832623532393035343930663434343737643032303764343138326638 -33386666656162393633636437656564326336343661643137323362653437323939663033316230 -61643437633665643439663566663232643162656139383236356538663139333162383665343766 -31326539313163333034666233316634396434316535313264313430313432313938383534663333 -33653165303864353638303833646134363362306632336133306430353165363063353735336464 -31303536333964333833343566383238636233366233656531343664346331386230316365626662 -30316635353262303962316364386263636531396636306534306464336536343138663933326236 -38653563633663356339633335376361326132356334626163616361383134306239643365343562 -34323031383638313336663937663564376362663534373464386635653164636430643266343766 -62326134623638343735613562366232653137613265653836326461386539646133303638613639 -31373465353533383237393933626333383839313736636132646564666639336530386537663239 -34653533643039633337353535393832343439346536336232373837363734366430336539373732 -66646339616366313834333562303466373962656435373530666466383339653064666532663236 -64393032623234623561313766343233643232303630613165383761323435313130303361346237 -62323430353333383539613536646439336466633435383134626530326531643135326264376662 -37393832643662386137613230633964396338616638313764373464616665316531396462326431 -36666338666536613666643032376331626132316433653036656330353834323865663865366535 -36313937346136353534373631396630633432313065666639623538343332636665313236336366 -35613330336136623331346636646134316265613863323730663837303032303131376435643131 -39353965376565356136613533623235303031646331663166396631336434623561643464656263 -64376636376337336661363133663139343935613033333266663439326239646434633131316637 -38643164383136343939663332653433376362363432333265623166643831303639376637366363 -37336361336461363437646234663330393830393861373162663463366533366637646530373838 -32633764373164646339653931653066333833363534663234663864336465333164353231663032 -36333666653633643362343334303035656539636663306333613964663837366338613565626635 -34393964303866313835373936376130626665303031376637396639353734373337336534376563 -30643261663235323831366537613434376566323837633136306663663331333863643037336133 -63623035376233393431373361653964323833396362646661343235343332636435356365346330 -39663533353631643062613839643761633866373539613536326132386336373731633239323362 -31383863333330366665393666363864613632636537363436613230623030353436373061633037 -36326633356533666438353034343966366136316331376431396536393261383633373131656530 -36336364653962303730393338313435316162393561383161646232663430616239623964623431 -38383563386266343935666463636533353361623965326232663937326462366433366534306534 -66323139346238343031313762353166646235313463363435393236633337303561333930376439 -37613630386636636634396437623866326639343863353134346438663033353232646566336164 -63376535363235653362623630636661636162313462303664393032326538613139653039653065 -38626234353034643330333861316665306665313933363234623639373765303065376632643233 -39633764346132356564333236306333373664366134363037633730653732643130616665343466 -62356138656331626437666465663132663562336365353834353632383062326164356161366138 -66663532373464316264366661343863363265616438323132646361346135653063303031323833 -33306231353231316165373137616134326163313430613963653532316331323738633639356565 -38343537616330376263636165396139396365643663393637626132306564613263323362663637 -64356665646532326466393733303130386230363831666265396635363630336331326662363864 -39383432346265376137636639613162643135343433303435336436376161303766383232656166 -32323462313432643330303039373239323939626266383232393433366232636134643636646431 -37376330343461393766643664393864633265393631383566396530393364346634626535373133 -61623931386433396630616463353236393630343838383865666466383632363631373136393930 -35633934393963626263646137623262666335306233356439633334313765353161653566613833 -32346637393530313730376564656434393737383037613664666338666435373066303338623930 -63333763633533373437363666616263646564613466613830336134353933626336343036646137 -32633265636134333438386330663764633432376633383930333835363362613738353464636330 -38613135326635643436346565633833623630623736623231306237313662666666653532333234 -62383866373939326335653461653238346336636636383961613534326336653030386534653934 -39323266373664346437613362373162643532636566623563303139353430343638656130303866 -63346537323431613434643865353865633331653233343839376237646337613566663130306537 -39343735626236653636393465633038323465656661626561656438373835343361363735386638 -39623830303864636333373534396562623630653362366166386264643438373435386438383831 -34633264613230643434313037323538346661346464393362613434303135313136376131666530 -36373866313739323962333061343836363661346334383033613534393666636631363262323238 -64313330316464336533663764386666313732616338393162393434613437623730613038356663 -39383236316664313731626330323039373937366638373461373736633239666438306437666666 -66316639346432316631616261626262666238346530353563356436353332313362353431646430 -64643366656466333363313135653331303764633839316239333630353665666465393263633333 -31306663326366643032653762646464396530643662303636393432373966393934396634343032 -31346365633933313436643937626634656535323431616231393866663334303632613237633666 -65396262363665353837366166633435326133356330383066613537653736656565306434303663 -38313837303965633236366533386661366464613262326162316365626534356237663035396264 -33356165623232323664363731383962616465653230373234323961646236356132383031346465 -61396136636137303933346462356339393236313361316465346631666432626434636366396531 -35313264323766663436666430313864303065313633396631656438666564313038386161313433 -66383734373435633838666463643636376366633334626130333730366639663961666630383234 -35343964363931663431373162653333613563366462333037363838623662303837626237396132 -36366162313766366237306337333338623937363165306433616530326435363063333938393939 -61373433666233393962633263386534333735303564326435623363663334313163326239653966 -30656132353938326530306663613637373838323432323762306263623136653937383465613038 -66346661646231623161373233653337353162636334376137373138613864353036663964336362 -64303362306239633339313666303636626264353833333333613132316261333830616232333662 -64623061633466336336623734616632303265363630356665303663636334656136636339353534 -65636564636462346133666438623461653630333833643230623632333266663066376461316661 -37623637356333386638343937613361343265663033326337666138393233646639656232333864 -37343733646432353564316237363964656532386561666538633737323739383766316337386137 -65656231336463623965653232323162363866373732656534376430356432653539393939343661 -30626130336436343033386637393334363237373737613761363633316533623033343731356338 -38363231303637323861376662356564323032316130356461653466336165363363383637313862 -35666338383164666135633734386432353137363464376538663465353330376237363333653566 -32376462383732656135333861323465396664353032393432373861353836376239346338393230 -39623532336232343761303235393963333065653935616230656236646330323465303263663236 -39336233303030656161343961353566626462616333643235623032623338646163346564383034 -63333439653936653731653337396231396562383565363837643138373864613338386164346262 -31613133303833323238343665353238323335623162353362646337336335323639363730386462 -61633966313435313165643537626330663864623762633330383730613131396463313231343837 -61623965393836303230626463363438343239363562643334653733343865313334383233343932 -33313930666136613766376435636432303466626565326434636561373565306532376239643735 -63613037616534656265663536373164643866653466326361313965383635626331636463383363 -35356533393461376132626431383839393132663161373831366130663964313839613566643165 -63353139316333666337636635303236643034623830353462626535623064653065316135343564 -64363963383230313765643262396330663938366631623834376532643937316236623865613665 -34653331333336333364396330646632326266626638316566363539313532323762383034323036 -38343838616264333737376432366535643339323734316563386439313339653763343139306565 -61383538613761353733653933316634316138313565393932623638636135303962366562623231 -38346365616434663362306138373634613334613037613961383032326661356163626238666233 -31653938633938366632326439333237663539383461646236306265326536626135343161396137 -34616636663639396534363431666464306538623632623133623133643864313839653865623266 -35643865366465346532386130353565633333613061396139383265663734323139663532376638 -38623966306639363938643564343330636363346261333166613938623936303966613037663365 -62376334613363323563363332356463376532323964333338623239326663343031373635663032 -33626432356662636130363761323862616131653630396164623435333061366564626236643936 -65316562633036366233303538373530343932616533346637656464346166306438396435666533 -37313766366636336535363536333061373264663163303162666330633061356464353164636239 -39623836663066623431343835623264373239363766323338353633336338363536643462666236 -35313031633561353066306336336362643863393730613938373539363831336266353266633036 -30313039336631396664383434316462643366643532353638396339353430303063303734343830 -31323761643235663061336237333866323139613533366463353433613832316232323938323331 -39356261636231353936313033363962353232663831316634653634323965313035663537323864 -33633138383431376131616462363337313035326339646661643730666337393266316562333138 -39616636323630356631653834313466393533333761666534333239306666356166363433326434 -33643566323438333934363236313633666465663236353034653439323338613433333538336535 -30326435383362343065356463373761333565396131336562636236353863613635636436326131 -37353238656337633135643430306532393530653032663437303139323463613136656563613633 -39303463656631336231643733323439376331653732376639303231333533373264623034316435 -64386635636565333961663936356131643536636235666433356338646531306535653366363362 -64336634396666303364663033383034616639646532333238336138343032376631326139646664 -37323432613664343032313838663561373735323938666339336133653131383263366665656537 -61363139356666303438316434626536323362306539613930353263663263353933353664393666 -37663763616635306665386163333034643465376130313861633435366666653535383234363438 -34326565373461343035616162316532333739386562383466626436366434306261363931653464 -63306238663038373631363834313261323337393263323362306636663765393535303433643562 -63616336383563343732653034316133343662333733323231623064386561623535376235336432 -38646537303136653939346335653031636266343132626262643439363564613132356634666435 -34626562336630363363393261643735353263653462353336383630366335643966396434363034 -65623233346335343637633564393466616365626361613134306362663333353730336264616664 -33656361323631663639303033663632663262313461663133646663633833383833636661336463 -37633266616639663830343333336362653838316363303637333535663136336633393339666263 -37313434303638616530336366303865333038326534656135613136613930376635363062353962 -38323339356235663965383633323665313932643632363132386666363239633666313566653063 -64643433363234396563356435383335623133613364313834396236376339633738326235363737 -31303566613461386434646233306261393835613739353831306164343164616432656466636139 -62393132636465663965633432353665643535643233313738323236383166653838663338323462 -61383035636632323863333361643333363238636436616562393465326230373164366261313564 -39343861333562373762623437613334396634613462373132386564616462333638373161653737 -38313366373864333064646133393561396537353333626462666361326532323239376431326330 -30373734346566663235616235316633636263373431326166623734303734383337643562666536 -61353130646537623936646639393439306639356130373431623431643834386632616165383133 -61666165396634613763306561356533663263386639643766306166303162376366363530396361 -65356538663638633261636332386262343236663731313932656339323239343938666662343064 -66353234346530366163656437356464376136663261343938636566333865313261396164316137 -38663238656365383131363239333537646333656136656465636464656530306137363436636231 -39366339663630653461383461323234363936346230643266393833356432396661356263333238 -38656530306133363733336335386238626132356237313334633634393634333937643631623963 -36326364306664613832363734336463303635373262653632356263336665353739373231316664 -34633036383439393339363364346165376261613339356236323330306432616665343265363564 -66666137346231383465353666633835393737313261643935656530356162613538363864343830 -35323865613563366130323132326238346338303761353433656532643131333962373066336233 -34303734396235303137616362653037333138306132393537646531616635636561343831366531 -36333932323234643237346266323466303736663665363362373465333139393934623939313535 -34316433323136316533313663393238663530386565376131366631393031643362346239316430 -64303636343765326465616464663234373137393137666635306261653466323733303962393461 -61356438316631333231636465653933343264356230313937636431336261306464316532343335 -32383732363866383531373136353331623566636332643332636237653737313064393566373837 -37333031643034343637333832363931343938303764313836373230353931613535313765393065 -30623234653039316461626635366337633265623236333963323934313065623732353362303338 -64663130383030613536383739646562636662656234363036613162666461373834333530363363 -62383166353333663232363737653639316235303535616536383962653039386464303435356538 -61653166633162616563336663323830383732666264393262323139303936626464373132303738 -30343061313535363838653930306164303438353736393862643461363333396539326633386236 -63623666343330363166373037636332313966303565303061663361386239353962666164383435 -64626663383735393431623436323862656262396461393663616530353132646234353838633537 -66313437306330653963613663373235383338383439636563643333323265626132326639386639 -38663762326666343935393664643761393236323266336265393033643734333439353538613062 -38626438663431366464343937316165313937323732643061643334643437353764333231646362 -65393566663530383133393430383237633638363363313963383230663434353635353364333830 -30306535353962633865346131333035353638336539336635643238383762303563323663616164 -63356630393664653637613935363536656239616239316337353730666632633234613331336136 -34383036616432306533663164316163303332313239646662656536333564643539343533383939 -32303935313565626661393334303564633136323164623965663364323966346636396532336535 -38373232616265346131633930333134666333663961383063666237653462623635323537393837 -30373438616561353430633139323163353138363735636530643630346566633631383931363638 -66633266643762626530613238343663326236363134316337653861376561383631663937373833 -32663063356164333731343263393837386332623665313135396665373436323366613331363333 -65393533353933643134643362366330653261313064663963336539653134396437383430313666 -33663234616461626630646563616561313139396630646461386338623266666434376335306435 -36643430663131373064313464633733356131386338643439653032353436656661356366626533 -35343434396361303634636438633036386563383664646561663566613339323865363136333230 -37616265613030633061376639386461383062353135666534346632353064353231 +62636131396363616337633131353962356639613766396163383633353937303334306661656662 +3734393736396462323634396236386339623336663033310a366563623930616331616333343337 +35613165633465653630626139393930333263323238353530663065613734646334663833323638 +3364613961656433360a633136376664356262633338356163363631656665623562653034376338 +33376661633533653766636135616261633133313338383238313335363461373064646261646262 +36383030393235653566666537373232666538616435356332373530323536386535666532656461 +32363638613131616137383133613966353232323537373938306232643738353163306336613238 +36396539623730333232653338393636333464353536623337323935613461396136313131326133 +35646561323165323431633039366435623965396363353535343165656631626166393234643639 +37396139376531653839646233313837373136613337326439373062393030393130623638633334 +30343266323938333333633064376632663737343332306533653232653837376637656166393935 +32353231333532326436383532646336356162333738333037336337363133396465363937343330 +38333335316663333232663966376234323261353036363262353362643061396533646139633539 +35343065653763393032346333373734376133613039396161343438666566613530653463623239 +63643762663933653333363665383634373762303462313064323336353862363530343831643336 +64653030386438326633356138316338353838316562643138353065623262373639326638646365 +38653637633731326364653834393132336637663135366230316437316365363366343530316236 +31616466383462626266316330303532346536653435643234646534373064313564343661383139 +33363864386266313563643264306537303066366463633062346237653963383131633436303234 +32613462616261623632343566653836623539666539643865656366313264616635616638386237 +65396135303739616635356332353336303861373965323834303261396162393235663963383963 +38376136616537396162333566326465323761353030393961623262336165373931633333353330 +30313137326661363137366265376132333063326130356334616537613330373664336462666137 +39366336356332613836343039623330343231613530373463316435323034653732333362336362 +65313763393863303935353430333430333632633036363932396238303464353465363032613536 +66663766356262313135653964636437396665323461386337303731326166386331363134373133 +65636436323336393034666638646632643336363031306163643266323333633031653734376662 +61313734666436343062373037336262633333356462303562346536663862393431373334323363 +32616663383135613933303430323030656130333264343231393164336536326464623964313131 +38366139393732376665373537376563656332626530636363613433666237653835306433363031 +62323734393933363064636464616132383636353965366365386261303530646233336266326366 +63663332656233393232376235373836306432373434316532356264613733643530373230656135 +32626463613663653336303064306563323133383834313936316366623931396537396165333563 +31633264663030643164353138653565363533373336333833386330313335636230353030613061 +39333462663965333565363730313731623538306436303337653361663035386130333665623463 +63643536653831393361336665616361653764356163363832656463363038343330636239356136 +33316238343538306538306564326337396633666166333863646462326164376365303836323530 +33636230376635323166343631373530623437366461393638383532656564303233383865626332 +65343063666462656236346634396464386437616332303437333932663966333162623463353733 +66646563313363633361393862616563353232656136376138306239383561386636333231373436 +30333336363831666530363132613862306334656133666231633063313263373336373562353238 +35323166366163343430323439646164646161623264663966643334653237366335666464373131 +37303663643638316334336665303562363266373130656230356336303531323161393365383430 +33366165613236363138636237363766366366363539643361323664613461383965616333346661 +30653436373030333232393366326435653061383963316632613163346165633335643731633961 +34396261663939636237383439613461303735393233653164363837333235303132626436626132 +65383661353261383437366235316533343664383033333966333531353131666530303439353464 +39396337373331663634333839656362653761306130363136353861653033303430323938663339 +64376636393935313633336464333930326432383164376139323533356438393761376462323764 +35313339363032373463613335666438306139396634386466323039663735643563333664636364 +63623636333837366165373637613431613962343966613934663065383333353761336331343466 +31393466393661336539383161613963313661313366316437393835303564393335336630366434 +34383434613637356232383863366465613133303964393033646239303932663563333535333931 +32326630336233663131613063363533333133336463366466313362333331633430353433336131 +39346639306630326336306537663262386264383938643362633762306364643435303533633635 +33333364333466393038353434363334633965336533313135396436336432393333653832363531 +38356633633632346263663935326264396632626263326663303861306363626134313761396233 +31396435653463303530633739323930396438366166366137386566346338366566393961643865 +33613365663863326331616163326666346633663136663330623331373439343330643261623665 +63656439653935653637353134623865633666343136396361363835363861366162323234363737 +62333135613539643261363462373534623631616438663035396162646161373665383364656438 +61653337316537336337313231326639386665636263383566303164353535643238386563366661 +63633037386536656332623331316361613064343833623766366665643232636265363731346538 +30353037393365353333393261666532303035393665356165633263663066623464313935343536 +61636462613738623861386534353332613734643534383430613564616235313432303566633963 +61616662633638336261326365373735343264393432323965316330373430656635323937346239 +61363433303235613630666331626361633938376564353831623664323334626233383133656235 +66643732613432376561636436306236663734316136643735376564383362356262666131373038 +66393931623637333237646332356531326533613932393232306461623235376664383334366562 +30356365613331646530616431306330363063623234396265633936343832663036333364646337 +32376138316533643839323766613838323265653632643430356234643665303666386664323031 +66343764643038666237343536393737623862663363666464396535336335616136373930336433 +37303862343834306266366335623036653738393234626434343337646338643631313662666634 +64383034333936643164323736303639383237623235356639383165646433303935343062396633 +32633436396530396330336238313466316334386531383962336165336461633836643963333164 +61663434326463623737333530353965366361616639396238386565393533363339333235643761 +64663230343964313435663335333039616165623466376638616534356638396538346234663436 +38636233643265353561356666306165616462373639306135393866386433366536373938353764 +64613637616431623963393534316631616439383161373135393161313332653134643237353037 +64313966636566343766326132376336623065363937303633363634356438396161303437656636 +34353562316663666430396630343665646165663061633235383232323363636637383465343538 +37303733653166316564666163333963656265623264656666613539666166636466333866316634 +32383466623962356338353334613166363932633330633836353037336139383732613766653030 +32373064363766303436643331363732326665646334316663323562376561353063663837346537 +39386531653762393263363261353136313961626663643865616266613834333664376664643761 +31393661393634383030643462643062333635366135313564666336613965623730313138646365 +62396565623636656335343737303732386161643537663236393831336336363764643762613164 +66396262363265386164346665383961333332346139363337396639343731346131373932653332 +62386564343639343930306164376432653961386165353333646134306231383334316366613337 +62326339366130653333643333616636653937323130396563636666373832383066343666376465 +37373932326538666664313534633661373035333765323733643064373865356338376130626137 +32373763373734383539313839666665653536356335373933626361646139303235656132343033 +30346435396637313234346436303964393863303735323864613038643136343237376138346334 +31333637623339343263646533383434393862626134303463663832613836373565373164326435 +62663461366639353537376664313463666437336264323165333734386362313430653839353437 +66373130613338333034333663366564616562363632336236363237653536663963326230633136 +30383635356362616263653637356234613537363364313437636662623038643537666431653533 +35383165366537663839633638663938383836616366313831643630306435616431643464393562 +30613634303266383535373965643534623338623466633365643839396564613236663631343439 +38346164653764383333393432343565326537313639336630366163653937343566303938653665 +32316262653630623638373964623430633466333835366439666530356535636630643063393562 +37643966346437396463383133306466336365303137313932376536666462313262343131623161 +64376261633964623761623666636134313038396564633436383839613730396139636530366333 +36346338313330353437343464383232666338613337336464653663613662383230643939623136 +63383163663062643436326166313432306166646264623933363935663862373737383434636234 +36336464353934383034373166373733343133353165623661643333663332303266643538666530 +35653063366463333961303761633763373266636639356239653636653061303865636332636632 +35633333623236313566376131333830353734303665383433366239353562666664643662323065 +35363164393138636135663462333131313632626535653239323463653634623339393061663861 +61626531383964373730396536663434306531383437333335316166663864653561643939643837 +30396137393665316239633330343161643562353735316330346565343366616536613330306664 +32343537363466386664353638623034616631346366633862353538386566616137653162303466 +38326531666130373864303235636261616565386264663335663862616265666366333739326631 +38323766663864353331396334613735613436363263653938646661623735393038393661613266 +37313932313434313735653330666639623730666462396433356364393931313632383634353334 +66346539663836636139393839653237623932333537653666333832626333396235383731373237 +33616634336662396538323436613166616339636235663933313437353737376539663438343164 +34383138363833616364393537333331396566643562346464373134313433333931343939336261 +32653231656265326662653639376365643234383265363466393864363562633066376664373261 +62343233633034633261613839333461336639643136656166326534383366343239323262303630 +36336265313233386361366139623663393638613538313265366462356463633862336364623962 +31336363653964303466626239356435666562386535663832373333353062343034646432383832 +35383831383331393962376665613634303634333864613066323530346134303934386564313666 +36656162343965306461306132626336353435643334303531633736613633646536353433626437 +63373033323434616539363035626632653663613838653232303039356633636264643432323236 +35313461616231613563393436633933363935306236316466313166636564323139393137323534 +34613432306237356663356434656331626161663138663665613663313263366162623561616234 +63663865623765643236343035353739633266623366393139333261353539313462306132326463 +61636665353861663838396236643563353134663831643061313234653830643864386131366562 +65643866633632353932313733363163373237333635623736666139393766393634626430333039 +63353536383138313165366362633230383137626364326333353037353736633664663738343766 +35653536626531393832346634666131333730643465656136653433373134306461383231613963 +63393366663137653635383830613239333337626432666436666363343637346331303731626137 +32326366643439333966616165636632396366383364313832356636643032306131396534653339 +39383732633132646464376233383139653865393061623763313865343964613031643965653939 +36633963633033313530666661363331353836666137386536396533306165646131363235303336 +32663537316164653935663034313833633135333837636331303363653864313231353734633461 +31383035376639393335346531626266323730363934393062366263313136343665303036393933 +62643965653932633334366363633763353862333335643265386533383062346266636639653064 +31613464393866323363393731363766363063303234323861363166373430383266663234336332 +61386561363562393736343939356434386464626134353634303931396662313665336531373235 +36366463623562393139383033303630663062323564626135613837623335633763613435353964 +30323161666264646630623532326431616135396133323838653736356338616339656638386464 +31363331303164373031326161353666613337653536656330653865653062316636656237376562 +34386431346330636464616136336531333331663865656266333964316261613265306434653038 +35333935626461353033323733326634373036663133326336373665383233343636396466313566 +37396462363764326361363561323633633537353165396362366166643362636639646536633065 +34643633383264663137363264663964626335643938333933306232363466346430666230353561 +37643836623262343664323664646161353230393435393033306536393531323166363739643432 +32396236363839313131306266383866323861343537373738326265326637636531626431373335 +65386564333961346633663633313737396565383431373265656131666638626462653335393330 +33356362613535313063316330306330656263353463313538643232663136636136333539363663 +30363066653633306537626664323339326565633134373831636633623930343635396362393561 +35303237616335653263643132386237386639373939363536356238656163343133613065376531 +66386361643465633938353438333730323933366465643533323738653762353437353136373134 +36636533313665633362323361373839633839653338633336373561386663343361313733393031 +34626566386635346439386564643261373363323466633839316564336137333636353364613062 +37343038373933316665376539396263393765613034306139393962346563353065316638613738 +66393739663736383265383735393331663739306462393939626661656534366665333132623831 +34623962393535353665343537643938353033343634646664323535393762366639376132626362 +34373230663937336161333934333935343564336237313837316363623665313564333134393133 +64653562313036303137316436633065643062303335643266366361316163306433376565363034 +63633830633937343530653537663465336563373435376366623537383165663033623539323333 +66313238363266653065623532616331623635643038353032303266373565646136646538623530 +31613038646331313738373634353737613435303964316161323938316261303237653362383233 +64616133623432353139323835366238633562336161626564633530623035336232346465386332 +64343036366430646161633461306661663564343132346564346633303934663539643239323536 +61386238663037373534336535396230613565303566626631323138643365656132333635613532 +34613633313633366638376662366464333664636538373133373939613565356637383863613035 +30353261643161633734386331623765326664616636333965363161343661613937356636363864 +38633132326237623964643734613165613133633832396633393432303535623237663864373135 +34643964636565353635663633623933643164303734353766643665313933343565373530333833 +36636165323030336562313764633039313165323630313665353366313135653636313338663233 +66663431346234333364666661643165653963616230376130663062616366383238666561326364 +36636530333633653763363435373534616435326533383030386164646130646164323262663330 +32323163623534613036316134366463616639633934396363653539343830643239396633333064 +31323264663866353530343033323066393332323161363031626136336531666566343332663230 +62653635313763313461376236663432306130383264313763356531646637363432396365613766 +37323935623031656438643330616563343632376230666236356534346466323533376631346564 +63333536313132393164356337643465373737613939666666363361303636333436306632626166 +63396637313235396236646237383162366531303133343962333639326139613231663737313532 +37373732383734646330326662636330646135666462343133663336666462643264313565313933 +31373035626564396665323235616138326539336133636232633034383262396466383138626264 +61663166356162616465353565646166636238376464383033303636623334666236396637633635 +30643861623632663632616130316630333234666439393030303961666239613530333937656563 +30356363313733393663376433346138383966616465343934663734666437653863663133643437 +64613236633736333237383834323338313431326634333730656536323138626438306231333035 +35373936646264363337613134396337613039366363323638656437656564623831666461373064 +66633232623666383738613834663938626438653430376534306164646363306663643362623137 +61376433636336346266393532363131613938356262303564393636613833643738646230323938 +32353763326631656561666533366236656134343638393435666437646231633463316232666563 +63653635393334336266653265326239626531323035653930323438393235633962386438306133 +34333537326530383363343139363461646339383064396637663465393961306139643761333538 +39346362663830373535386665386533643435666438656638373861376537383538366331323461 +66363138326432363432343335636434656139386161313162643838343566356264313734613630 +64366162393932313131306533633666393564613835363664306639343161643339316537346338 +64303764366136303737666464393463343261383538333563313164343666383132333761306463 +62656138633939383239343533356136383735663834323763623437386434313333306230326237 +38373666376332646635366233646237373431336531366432323130353034666265303236363937 +36356536653465326139306635633032326535373734333664656339386266363365303139623463 +38396664613639613737343062326538653835373861393939656636366337366134393236346463 +33306462633639626463326230656338636664613730373731323531636337333062646233306139 +61303661313764623565383265353061616638663636336635356364383933313363663330306638 +39613637363935626363633635333533323735623963346334303661353030313066633636396361 +65303738383131363237653866636637346464336334646362326538303232333661623038366332 +36323736333538373931616638303030613765326165663066333131373734373766343066626131 +32626462653538376431373635363835643036376230623061633963393065323262353931613165 +39633136656666653231353036353034333634646266386533316232303462663161613239383931 +64623034333137376134373235626165356466323931333131376238306536633861366430616362 +63636233653339376665346635656239306161663538656166363563326564643661316563346666 +38666161366430663234396237316534323263363436613435643436376535643430323432333639 +32373436353936623966626263653037366263646232336665626533343366366530393535663733 +32616639386433303032376565626535393466356136316662626364376666373934613166303233 +39336136666437323766336430653433393134656230663961666239306639323039396432653936 +65363736343637636538646130616362373965376437333762383138393134353332663734326565 +31303136613739643965623437643663653339666638363861343164663538313234313330663738 +37646362636130663431633762323634613964666166323962346234643765336639613934393938 +64323239343832326564333731663961626661643936643233336662303335663861653331326161 +64396564316465353163633066383562643636363562376136363733646439356530643432336139 +63366265353435323565333161386563636134306261653366623036396266356137333364343930 +30313465313834623530663436623333663262646331653832656266353834323463666332643130 +32656539393838393766643962396133333038343166336166336539333434363433656265313339 +31613238643339656130396461306134646666386430313636376633333465356230333433303666 +37313534343037336432373639303534633235623264363064653635333837656132643665396132 +66336463313139336163623637383164643237636134653837663062303238633237306562616161 +62613536653036333431346131363864393730333432383632333034646361373731316431343662 +39656264323965653738316264636361343361303466326336663438643430366235643165323632 +34313331353530386266 diff --git a/ansible/infra/402_public_monitoring.yml b/ansible/infra/402_public_monitoring.yml index d572171..2dc4689 100644 --- a/ansible/infra/402_public_monitoring.yml +++ b/ansible/infra/402_public_monitoring.yml @@ -5,7 +5,8 @@ # is the right way round for them: all three are about how the outside world # sees us, so they must be measured from outside. Gatus polls from the # observability host and needs nothing installed anywhere else - there is no -# script, no timer and no token, because nothing is reporting in. +# script, no timer and no push token, because nothing is reporting in. (The +# provider-balance check does carry an API credential - see below.) # # That also means these have no heartbeat. A heartbeat answers "did the thing # that was supposed to report in do so"; when Gatus does the checking itself, @@ -150,9 +151,50 @@ 'minimum-reminder-interval': '6h'}]}] }}" loop: "{{ public_tcp }}" + # ── Provider balance ───────────────────────────────────────────────────── + # 99stack is prepaid: when the balance runs out the servers stop. The one + # pulled check here that carries a credential - unlike everything above, + # the API will not answer anonymously (403 "Authorization token is + # missing"). + # + # The condition is RUNWAY, not balance. billing.secondsdue is the + # provider's own "seconds until you run out at current usage", so it + # follows the spend as servers come and go; a fixed balance threshold + # would be one more number written down once and left behind. 336h of + # runway, same as domain expiry, because topping up is a manual act too. + # + # The token is account-wide - 99stack has no read-only scope - so it can + # create and destroy servers. It lands in Gatus's endpoints file (0640 + # root:gatus), the same place as the push tokens. + - name: Require the 99stack API token + ansible.builtin.assert: + that: + - ninetyninestack_api_token is defined + - ninetyninestack_api_token | length > 0 + fail_msg: >- + ninetyninestack_api_token must be in group_vars/all/vault.yml - an + API token from the 99stack control panel. + quiet: true + + - name: Build the provider endpoints + ansible.builtin.set_fact: + provider_endpoints: + - name: 99stack-balance + group: provider + url: https://api.99stack.com/v1.5/account/info + interval: 6h + headers: + Authorization: "Bearer {{ ninetyninestack_api_token }}" + conditions: + - "[STATUS] == 200" + - "[BODY].billing.secondsdue > {{ 14 * 86400 }}" + alerts: + - {type: signal, failure-threshold: 1, success-threshold: 1, + send-on-resolved: true, minimum-reminder-interval: 24h} + - name: Register the public-facing endpoints ansible.builtin.include_role: name: gatus_endpoint vars: gatus_endpoint_name: public - gatus_endpoint_pulled: "{{ domain_endpoints + dns_endpoints + http_endpoints + tcp_endpoints }}" + gatus_endpoint_pulled: "{{ domain_endpoints + dns_endpoints + http_endpoints + tcp_endpoints + provider_endpoints }}" diff --git a/ansible/roles/gatus_endpoint/defaults/main.yml b/ansible/roles/gatus_endpoint/defaults/main.yml index 90a2c19..4da927a 100644 --- a/ansible/roles/gatus_endpoint/defaults/main.yml +++ b/ansible/roles/gatus_endpoint/defaults/main.yml @@ -12,6 +12,7 @@ gatus_endpoint_name: "" # # A DNS check adds `dns: {query-type, query-name}` - and note that for those, # `url` is the RESOLVER to ask, not the name being looked up. +# An API behind a credential adds `headers: {Authorization: ...}`. # A domain-expiry check is just `url: ` with a [DOMAIN_EXPIRATION] # condition; it uses WHOIS/RDAP and needs no scheme. gatus_endpoint_pulled: [] diff --git a/ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2 b/ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2 index 680691e..34e63e4 100644 --- a/ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2 +++ b/ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2 @@ -41,6 +41,10 @@ endpoints: {% if e.client is defined %} client: {{ e.client | to_nice_yaml(indent=2) | indent(6, true) }} +{% endif %} +{% if e.headers is defined %} + headers: +{{ e.headers | to_nice_yaml(indent=2) | indent(6, true) }} {% endif %} conditions: {% for c in e.conditions %} From 5447e03d1fa6e1b95ce42fae7843ccd0acc1ad18 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 19 Oct 2026 22:10:26 +0200 Subject: [PATCH 15/15] arbret: our side of running it on vipy prd-arbret was folded into vipy. arbret itself (app, worker, Postgres, timers, its backup dump) is deployed by its owner's own repo; this repo owns the shared host's side of it. arbret_handoff.md documents the split, the Gatus push API and the backup contract. - services/arbret: arbret.com -> localhost:3100, www redirects to the apex, port 4117 allowed on tailscale0 only, /opt/backups/arbret pre-created to the contract so the pull job is green before the first dump. - headscale ACL: framie may reach prd-vipy:4117. - vipy: arbret-server, arbret-worker and postgresql in the unit checks. - 402: DNS and HTTP checks for arbret.com, www, and /api/health with [BODY].status == ok. Entries may now name an fqdn outside root_domain and add conditions. - 403: probes arbret-analytics (hourly, 130m window) and arbret-beancount (daily, 26h), pushing with the dedicated arbret token. - backups: pull from backup-pull@prd-vipy:/opt/backups/arbret/, and a backup-dump_arbret endpoint. Co-Authored-By: Claude Opus 5.5 (1M context) --- ansible/host_vars/vipy/main.yml | 5 + ansible/infra/402_public_monitoring.yml | 21 +- .../infra/403_service_probe_registration.yml | 9 + ansible/playbooks/backups.yml | 9 +- .../arbret/deploy_arbret_edge_playbook.yml | 59 +++ .../services/headscale/templates/acl.json.j2 | 2 +- ansible/site.yml | 1 + arbret_handoff.md | 456 ++++++++++++++++++ 8 files changed, 554 insertions(+), 8 deletions(-) create mode 100644 ansible/services/arbret/deploy_arbret_edge_playbook.yml create mode 100644 arbret_handoff.md diff --git a/ansible/host_vars/vipy/main.yml b/ansible/host_vars/vipy/main.yml index 3b077df..93415da 100644 --- a/ansible/host_vars/vipy/main.yml +++ b/ansible/host_vars/vipy/main.yml @@ -13,3 +13,8 @@ monitored_services: - lnbits - caddy - phoenixd + # Deployed by the arbret repo, not this one (see arbret_handoff.md); listed + # here because "what runs on this host" is still a property of the host. + - arbret-server + - arbret-worker + - postgresql diff --git a/ansible/infra/402_public_monitoring.yml b/ansible/infra/402_public_monitoring.yml index 2dc4689..3ff5b23 100644 --- a/ansible/infra/402_public_monitoring.yml +++ b/ansible/infra/402_public_monitoring.yml @@ -35,6 +35,11 @@ - {sub: "{{ subdomains.monero_explorer }}", host: vipy} - {sub: "{{ subdomains.music }}", host: vipy} - {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy} + # Not under root_domain, so named in full. arbret.com's zone is NOT in + # Porkbun (mynymdns) - this is the check that would have caught it still + # pointing at prd-arbret's recycled IP after the 2026-10-17 rebuild. + - {fqdn: arbret.com, host: vipy} + - {fqdn: www.arbret.com, host: vipy} # A public resolver on purpose: this must test what the internet sees, not # what a local cache or the tailnet's MagicDNS happens to answer. @@ -68,6 +73,13 @@ # tailnet hop, gunicorn, and the database and bucket behind it (503 if # either is down). "/" would only prove the login redirect works. - {name: jars-staging, sub: "{{ subdomains.jar_lid_archive_staging }}", path: "/health/", status: 200} + # arbret (runs on vipy since 2026-10-17; deployed by the arbret repo, not + # this one). /api/health answers 200 + "status":"ok" when healthy and 503 + # "degraded" otherwise, DB down included - per the arbret owner. The body + # condition guards against a 200 that is not actually healthy. + - {name: arbret, fqdn: arbret.com, path: "/", status: 200} + - {name: arbret-health, fqdn: arbret.com, path: "/api/health", status: 200, + conditions: ['[BODY].status == ok']} # Ports published from the edge host by socket_proxy. public_tcp: @@ -107,11 +119,11 @@ - name: Build the DNS endpoints ansible.builtin.set_fact: dns_endpoints: "{{ dns_endpoints | default([]) + [{ - 'name': item.sub ~ '.' ~ root_domain, + 'name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain), 'group': 'dns', 'url': dns_resolver, 'interval': item.interval | default('6h'), - 'dns': {'query-type': 'A', 'query-name': item.sub ~ '.' ~ root_domain}, + 'dns': {'query-type': 'A', 'query-name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain)}, 'conditions': ['[DNS_RCODE] == NOERROR', '[BODY] == ' ~ hostvars[item.host].ansible_host], 'alerts': [{'type': 'signal', 'failure-threshold': 1, @@ -128,10 +140,11 @@ http_endpoints: "{{ http_endpoints | default([]) + [{ 'name': item.name, 'group': 'public', - 'url': 'https://' ~ item.sub ~ '.' ~ root_domain ~ item.path, + 'url': 'https://' ~ (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain) ~ item.path, 'interval': item.interval | default('5m'), 'conditions': ['[STATUS] == ' ~ item.status, - '[CERTIFICATE_EXPIRATION] > 168h'], + '[CERTIFICATE_EXPIRATION] > 168h'] + + item.conditions | default([]), 'alerts': [{'type': 'signal', 'failure-threshold': item.threshold | default(3), 'success-threshold': 2, 'send-on-resolved': true, 'minimum-reminder-interval': '6h'}]}] }}" diff --git a/ansible/infra/403_service_probe_registration.yml b/ansible/infra/403_service_probe_registration.yml index a46682a..f8cbb91 100644 --- a/ansible/infra/403_service_probe_registration.yml +++ b/ansible/infra/403_service_probe_registration.yml @@ -59,6 +59,15 @@ - {name: xmrblocks, host: monerod_box_local} - {name: navidrome, host: music_box_local} - {name: jar-lid-archive-staging, host: jar_lid_staging_local} + # arbret (on vipy, deployed by the arbret repo). `host` here is the KEY in + # gatus_push_tokens, and arbret has a dedicated `arbret` token rather than + # vipy's - so it cannot write vipy's own results, and vice versa. + # Windows are two runs plus slack, as for the 5m probes above: + # analytics hourly at :00 -> 130m + # beancount daily at 03:00 -> 26h + # Cadence lives in the arbret repo; tell us if it changes. + - {name: arbret-analytics, host: arbret, heartbeat: '130m'} + - {name: arbret-beancount, host: arbret, heartbeat: '26h'} # The alert every probe gets. failure-threshold is 1 - the heartbeat window # is the tolerance, see the note above - unless a probe sets diff --git a/ansible/playbooks/backups.yml b/ansible/playbooks/backups.yml index 875f10a..791d51d 100644 --- a/ansible/playbooks/backups.yml +++ b/ansible/playbooks/backups.yml @@ -12,8 +12,10 @@ backup_store_check_push_base: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints" backup_store_check_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" backup_store_sources: + # On vipy since 2026-10-17 (prd-arbret is gone). The arbret repo writes + # the dump to the backup_source contract - see arbret_handoff.md. - name: arbret - source: "arbret@prd-arbret:/opt/arbret/backups/" + source: "backup-pull@prd-vipy:/opt/backups/arbret/" retention_days: 90 - name: headscale source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/" @@ -52,8 +54,8 @@ # LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it # cannot tell you whether the dump broke or the pull did. # -# arbret has no dump endpoint: prd-arbret lives in [arbret], which `managed` -# deliberately excludes, so nothing of ours runs there. It is store-checked only. +# arbret's dump is written by the arbret repo, not by backup_source, and it +# pushes with its own `arbret` token (the `host` below is the token key). # ───────────────────────────────────────────────────────────────────────────── - name: Register the backup checks with Gatus hosts: observability @@ -67,6 +69,7 @@ - {name: lnbits, host: vipy} - {name: forgejo, host: vipy} - {name: jar-lid-archive-staging, host: jar_lid_staging_local} + - {name: arbret, host: arbret} store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo, jar-lid-archive-staging] tasks: diff --git a/ansible/services/arbret/deploy_arbret_edge_playbook.yml b/ansible/services/arbret/deploy_arbret_edge_playbook.yml new file mode 100644 index 0000000..3449e95 --- /dev/null +++ b/ansible/services/arbret/deploy_arbret_edge_playbook.yml @@ -0,0 +1,59 @@ +--- +# arbret's footprint on the edge host - and ONLY that. +# +# arbret itself (app, worker, Postgres, timers, its backup dump) is deployed by +# the arbret owner's own Ansible repo since prd-arbret was merged into vipy on +# 2026-10-17. This playbook owns the parts of the shared host that are ours: +# the public front door, the tailnet port, and the backup directory contract. +# See arbret_handoff.md at the repo root for the division of labour. +- name: Publish arbret on the edge host + hosts: edge + become: yes + vars: + # Local port the arbret app binds on 127.0.0.1, per the arbret owner. + arbret_port: 3100 + arbret_domain: arbret.com + # Admin port reachable from the tailnet only (ACL in headscale's acl.json.j2). + arbret_tailnet_port: 4117 + + tasks: + - name: Publish arbret through Caddy + ansible.builtin.include_role: + name: caddy_site + vars: + caddy_site_name: arbret + caddy_site_domain: "{{ arbret_domain }}" + caddy_site_upstream: "localhost:{{ arbret_port }}" + + - name: Redirect www to the apex + ansible.builtin.include_role: + name: caddy_site + vars: + caddy_site_name: arbret-www + caddy_site_domain: "www.{{ arbret_domain }}" + caddy_site_body: | + redir https://{{ arbret_domain }}{uri} permanent + + # tailscale0 only. UFW's default-deny applies to the tailnet interface too, + # so the headscale ACL alone does not open it - and this must never be + # reachable on the public interface. + - name: Allow the arbret admin port from the tailnet + community.general.ufw: + rule: allow + interface: tailscale0 + direction: in + port: "{{ arbret_tailnet_port | string }}" + proto: tcp + comment: "arbret admin, tailnet only" + + # The backup contract directory (arbret_handoff.md section 3.3). The arbret + # repo creates it identically; creating it here too means small-backups-box + # can pull - an empty dir - before arbret's first dump, so one missing + # source does not turn the whole pull job red. + - name: Ensure the arbret backup directory exists + ansible.builtin.file: + path: /opt/backups/arbret + state: directory + owner: root + group: backup-pull + mode: "0750" diff --git a/ansible/services/headscale/templates/acl.json.j2 b/ansible/services/headscale/templates/acl.json.j2 index 2362a1f..710c7a8 100644 --- a/ansible/services/headscale/templates/acl.json.j2 +++ b/ansible/services/headscale/templates/acl.json.j2 @@ -26,7 +26,7 @@ "dst": [ "*:22", "tag:arbret-staging-box:4117", - "tag:prd-arbret:4117", + "tag:prd-vipy:4117", "tag:nodito:8006" ], "src": [ diff --git a/ansible/site.yml b/ansible/site.yml index 28e620e..7867397 100644 --- a/ansible/site.yml +++ b/ansible/site.yml @@ -65,6 +65,7 @@ - import_playbook: services/forgejo/deploy_forgejo_playbook.yml - import_playbook: services/lnbits/deploy_lnbits_playbook.yml - import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml +- import_playbook: services/arbret/deploy_arbret_edge_playbook.yml # ── Backups: each source dumps itself, the box pulls ──────────────────────── - import_playbook: services/headscale/setup_backup_headscale.yml diff --git a/arbret_handoff.md b/arbret_handoff.md new file mode 100644 index 0000000..5503a50 --- /dev/null +++ b/arbret_handoff.md @@ -0,0 +1,456 @@ +# arbret on prd-vipy: monitoring and backups + +Handoff for the arbret owner. It covers what changed when prd-arbret was lost, and what +your Ansible repo needs to do so that arbret is **monitored by Gatus** and **backed up to +small-backups-box** from its new home. + +Written 2026-10-17, during the rebuild. + +--- + +## 0. What changed, in one screen + +| | Before | Now | +|---|---|---| +| Host | `prd-arbret`, a droplet of its own | **`prd-vipy`**, shared with the personal_infra services | +| Monitoring | Uptime Kuma (push URLs on `uptime.contrapeso.xyz`) | **Gatus** at `https://status.contrapeso.xyz`, running on `prd-spacey`. Uptime Kuma is gone for good. | +| Backup destination | small-backups-box pulls `arbret@prd-arbret:/opt/arbret/backups/` | small-backups-box pulls **`backup-pull@prd-vipy:/opt/backups/arbret/`** | +| TLS / front door | your own, on prd-arbret | **our Caddy on prd-vipy** (it owns 80/443) | + +Who does what: + +| Area | **You** (arbret repo) | **Us** (personal_infra repo) | +|---|---|---| +| App, workers, Postgres, analytics | deploy and run them | — | +| Public HTTPS for `arbret.com` | tell us your local port | Caddy vhost, certificates | +| Gatus endpoint definitions | — | all of them: Gatus config lives on prd-spacey, which you don't touch | +| Pulled checks (HTTPS, `/api/health`) | keep `/api/health` honest | define and run them | +| systemd unit checks (`arbret-server`, `arbret-worker`, `postgresql`) | keep the unit names stable | add them to vipy's monitored units | +| Host liveness and disk | — | already covered for prd-vipy | +| Analytics timer result | **push it** (§2.4) | register the endpoint | +| Backup dump | **produce it to the contract** (§3) | pull it, retain it 90 days, check freshness | +| Backup dump result | **push it** (§3.5) | register the endpoint | + +Your work comes down to three things: + +1. Live on a shared host without stepping on it (§1). +2. Push two results to Gatus: analytics and backup. +3. Write the dump to a new place, in a fixed shape. + +--- + +## 1. Living on a shared host + +prd-vipy also runs Vaultwarden, Forgejo, LNbits, phoenixd and a reverse proxy for a +dozen other services. Our Ansible converges it regularly. Anything below that your +playbooks change will be reverted by ours, or will break ours. + +### Owned by personal_infra: do not manage these + +- **Ports 80 and 443 and the Caddy install** (`/etc/caddy/**`). Don't install nginx, + certbot or a second Caddy, and don't bind 80/443. Give us a port and we write the vhost + (see below). +- **UFW and fail2ban.** The default policy is deny-incoming, with only SSH and 80/443 + open. Don't disable UFW or reset its rules. If you need a port reachable from outside, + ask us. +- **Users `counterweight` and `backup-pull`**, and their `authorized_keys`. In + particular **don't create, modify or authorise keys for `backup-pull`**: we create it + and it is shared by every backup on this host (§3). +- **`/opt/backups/`** (the parent directory, `root:root 0755`). You own exactly + `/opt/backups/arbret/` and nothing beside it. +- **Tailscale/Headscale membership** of the host, and the tailnet ACLs. +- **Docker.** It's installed for our services. You don't need it (arbret is native + systemd); if you start using it, don't touch `/etc/docker/daemon.json`. +- systemd unit names we use: `caddy`, `forgejo`, `lnbits`, `phoenixd`, + `*-backup.{service,timer}` for vaultwarden/lnbits/forgejo, `liveness-healthcheck`, + `disk-usage-healthcheck`, `service-health-healthcheck`. Prefix all of yours with + `arbret-` and there's no risk of collision. + +### Free for you + +- PostgreSQL. We run no Postgres on this host, so the system `postgresql` package and + cluster are yours. +- `/opt/arbret/`, `/etc/arbret/`, the `arbret` system user, `arbret-*` units and timers. + +### Local ports already taken + +`22, 80, 443, 3000, 7657, 8222, 8333, 8765, 9740, 23334, 50002`. Bind your app to +`127.0.0.1:` on anything else and tell us the port. Our Caddy then serves +`arbret.com` with a Let's Encrypt certificate and reverse-proxies to it, setting +`X-Forwarded-For`, `X-Forwarded-Proto` and `X-Forwarded-Host`. Configure the app to +trust proxy headers from `127.0.0.1` only. + +Port `4117` used to be reachable from one admin laptop over the tailnet. If that's still +needed, say so and we'll add the ACL for prd-vipy. Don't open it in UFW yourself. + +### The host itself + +Debian 13, clock in **UTC**, tailnet name `prd-vipy`. + +--- + +## 2. Monitoring with Gatus + +### 2.1 How it works here + +Gatus runs on prd-spacey and answers at `https://status.contrapeso.xyz`. Alerts go to +Signal. There are two kinds of check: + +- **Pulled.** Gatus makes the request itself (HTTP, TCP, DNS) every N seconds and + evaluates conditions. Nothing is installed on your side. This is the right kind for + anything with a reachable URL. +- **Pushed ("external endpoints").** Something on the host `POST`s a result to Gatus, + and Gatus alarms if a result is `false` **or if nothing arrives within the heartbeat + window**. This is the right kind for things with no URL: timers, jobs, internal state. + The heartbeat is what makes a dead timer visible: silence is a failure. + +Everything about an endpoint (name, group, token, heartbeat, alert rules) is defined in +**our** repo, because Gatus's config files live on prd-spacey. Your side of a push check +is just an HTTP call. + +### 2.2 The push API + +``` +POST https://status.contrapeso.xyz/api/v1/endpoints/{KEY}/external?success={true|false}&error={message} +Authorization: Bearer {TOKEN} +``` + +- **`KEY`** is `sanitize(group) + "_" + sanitize(name)`, where sanitize lowercases and + turns any of `/ _ . , space # + &` into `-`. Use the exact keys in the table below. +- **`success`**: `true` or `false`. +- **`error`**: free text, URL-encoded. It's shown on the dashboard and in the Signal + alert, so on success it's fine to send something useful like `ok 31M`. +- **`TOKEN`**: a bearer token we give you (§4). It's dedicated to arbret: it can only + write arbret's endpoints, and nobody else's token can write yours. + +Responses: + +| HTTP | Meaning | +|---|---| +| `200` | recorded | +| `401` | wrong token for that key | +| `404` | key not registered (yet). We register before you deploy. | + +Test it by hand once you have the token: + +```bash +curl -i -X POST -H "Authorization: Bearer $TOKEN" \ + "https://status.contrapeso.xyz/api/v1/endpoints/probe_arbret-analytics/external?success=true&error=manual%20test" +``` + +### 2.3 Your old Uptime Kuma monitors, mapped + +| Old monitor (prod) | New check | Gatus key | Kind | Who | +|---|---|---|---|---| +| `https`: front door returns 200 | `GET https://arbret.com/` → 200 | `public_arbret` | pulled, 5 min | us | +| `health`: `/api/health` returns ok | `GET https://arbret.com/api/health` → 200 and `[BODY].status == ok` | `public_arbret-health` | pulled, 5 min | us | +| `arbret-server`: unit active | systemd unit check on prd-vipy | `services_vipy-arbret-server` | push, every 5 min, by our host script | us | +| `arbret-worker`: unit active | same | `services_vipy-arbret-worker` | same | us | +| `postgresql`: unit active | same | `services_vipy-postgresql` | same | us | +| `arbret-analytics`: timer ran OK | your analytics run reports its result | `probe_arbret-analytics` | **push, by you**, hourly; heartbeat 130m | **you** (§2.4) | +| (new) beancount: daily job ran OK | your beancount run reports its result | `probe_arbret-beancount` | **push, by you**, daily 03:00; heartbeat 26h | **you** (§2.4, same hook) | +| `arbret-backup`: timer ran OK | your backup run reports its result | `backup-dump_arbret` | **push, by you** | **you** (§3.5) | +| prd-arbret liveness / disk | covered by prd-vipy's own host checks | `liveness_vipy`, `disk_vipy` | push | us | + +We also check: + +- `backup-store_arbret`: the backup **arrived** on small-backups-box, is fresh (<26h, + read from the filename timestamp), non-empty, and not suspiciously smaller than the + previous one. +- `arbret.com` resolves to prd-vipy, checked every 6 hours. +- `arbret.com` registration expiry, alarming 14 days ahead. This already existed. + +**Delete, don't port, every Uptime Kuma push script, timer and URL in your repo.** The +endpoint no longer exists. A leftover timer will just fail silently every two minutes +forever. + +The pulled health check relies on `/api/health` returning HTTP 200 with a JSON body that +has `"status": "ok"` when healthy, and **a non-200 when it isn't**. If the body differs, +tell us the exact shape. A health endpoint that checks the database is worth more than +one that only proves the process answers. + +### 2.4 What you implement: the analytics push + +The cleanest hook is `ExecStopPost=` on the oneshot service the timer runs. systemd runs +it after every run, success or failure, and sets `$SERVICE_RESULT` and `$EXIT_STATUS` +for it. So no wrapper script is needed, and the run itself stays untouched. + +`/etc/arbret/gatus.env` (`root:root 0600`): + +``` +GATUS_PUSH_BASE=https://status.contrapeso.xyz/api/v1/endpoints +GATUS_PUSH_TOKEN= +``` + +`/usr/local/bin/arbret-gatus-push` (`root:root 0755`): + +```bash +#!/usr/bin/env bash +# Usage: arbret-gatus-push +# Meant for ExecStopPost=, where systemd provides SERVICE_RESULT / EXIT_STATUS. +# Can also be called by hand: SERVICE_RESULT=success arbret-gatus-push +set -uo pipefail +KEY="$1" +: "${GATUS_PUSH_BASE:?}" "${GATUS_PUSH_TOKEN:?}" + +if [ "${SERVICE_RESULT:-}" = "success" ]; then + success=true; msg="ok" +else + success=false; msg="${SERVICE_RESULT:-unknown} exit=${EXIT_STATUS:-?}" +fi +enc=$(printf '%s' "$msg" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g; s/=/%3D/g') + +# Never let a monitoring hiccup fail the job itself. +curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \ + -H "Authorization: Bearer ${GATUS_PUSH_TOKEN}" \ + "${GATUS_PUSH_BASE}/${KEY}/external?success=${success}&error=${enc}" || true +exit 0 +``` + +In the analytics service unit: + +```ini +[Service] +Type=oneshot +EnvironmentFile=/etc/arbret/gatus.env +ExecStart=...your existing command... +ExecStopPost=/usr/local/bin/arbret-gatus-push probe_arbret-analytics +``` + +Two things to get right: + +- `EnvironmentFile=` puts the token in the unit's environment, **not** in the unit file, + so the unit file can stay world-readable while the env file is `0600`. +- **Tell us the timer's schedule.** We set the heartbeat to the interval plus slack: a + 5-minute job gets an 11-minute window, a daily job gets 26 hours. If the schedule + changes later, tell us, or Gatus will either alarm on a healthy job or sleep through a + dead one. + +As Ansible, this is two templated files, an `EnvironmentFile=` line and an +`ExecStopPost=` line. It's small enough to live in the role that deploys the analytics +timer, rather than becoming a role of its own. + +### 2.5 Staging (arbret-staging-box) + +arbret-staging-box is a VM in our inventory, so its liveness and disk checks already +exist. The same mapping applies with these keys: + +| Check | Key | Who | +|---|---|---| +| front door / health | `public_arbret-staging`, `public_arbret-staging-health` | us | +| units | `services_arbret-staging-local-arbret-server` (and `-worker`, `-postgresql`) | us | +| analytics | `probe_arbret-staging-analytics` | you, with the **staging** token | +| backup dump | `backup-dump_arbret-staging` | you, with the staging token. Only if staging should be backed up; tell us. | + +The staging vhost (`arbretstaging.contrapeso.xyz`) lived on the old prd-vipy and was not +in either repo. We're recreating it on our side. + +--- + +## 3. Backups + +### 3.1 The pipeline + +``` + 02:00 UTC prd-vipy your timer: pg_dump | age -r -> /opt/backups/arbret/arbret_.dump.age + your script: POST backup-dump_arbret (success/failure) + 04:00 UTC small-backups-box rsync backup-pull@prd-vipy:/opt/backups/arbret/ -> ~/backups/arbret/ (keeps 90 days) + 05:30 UTC small-backups-box check-backups.sh -> POST backup-store_arbret (then every 6 h) +``` + +- **The box only pulls.** Nothing on prd-vipy can reach into the backup box. The pull + logs in as `backup-pull`, an unprivileged account with a `restrict`ed key that can + read the backup directories and nothing else. +- **The source encrypts.** Only the age **public** key is on prd-vipy, so the host can't + decrypt its own backups. The private key lives offline with the infra owner. Plaintext + never touches disk or the network. +- **Retention is two-tier.** Keep about 7 days locally (the disk is shared). The box + keeps 90. + +Your existing dump role already produces the right **artefact**: a Postgres custom-format +dump, age-encrypted, named `arbret_YYYYmmdd_HHMMSS.dump.age`. The last pre-incident one +is `arbret_20261016_020001.dump.age`. Only its **location, ownership and reporting** +change. + +### 3.2 What changes from prd-arbret + +| | prd-arbret | prd-vipy | +|---|---|---| +| Directory | `/opt/arbret/backups/` | **`/opt/backups/arbret/`** | +| Account the box pulls as | `arbret` (your user, with our key authorised) | **`backup-pull`** (ours, already exists) | +| SSH key for the pull | you authorised it | **nothing to do.** Remove that task, and don't authorise anything for `arbret` or `backup-pull`. | +| Result reporting | Uptime Kuma push | Gatus push to `backup-dump_arbret` | + +### 3.3 The contract + +The checks on the box enforce these, and some failures are silent, so read every line. + +1. **Directory:** `/opt/backups/arbret/`, owner `root`, group `backup-pull`, mode `0750`. + Don't change `/opt/backups/` itself (`root:root 0755`, ours). The `backup-pull` group + exists before your playbook runs; if it doesn't, fail loudly rather than creating it. +2. **Filename:** `arbret_YYYYmmdd_HHMMSS.dump.age`, timestamp in the host's local time + (UTC). The prefix must be exactly `arbret_`, because the box's prune and freshness + checks glob on `arbret_*`, and it reads the **dump time from the filename**. +3. **Encryption:** `age -r ` in the pipe (§4). Nothing unencrypted is ever + written to disk. +4. **Atomicity:** write `.partial`, set ownership and mode, then `mv` it to the + final name. The pull skips `*.partial`. A half-written file under the final name + would be pulled and kept for 90 days as if it were a backup. +5. **File ownership:** `root:backup-pull`, mode `0640`, on both the `.partial` and the + final file. If the pull account can't read a file, **rsync fails for the whole + directory**, so one bad file blocks every later night. +6. **`set -euo pipefail`.** Without `pipefail`, a failing `pg_dump` piped into a + succeeding `age` exits 0 and produces a tiny, valid-looking, useless artefact. +7. **Timing:** finished before 04:00 UTC. 02:00 is good. +8. **Local prune:** delete `arbret_*.dump.age` older than ~7 days, plus stale + `arbret_*.partial` files at the start of each run. Put nothing else in the directory: + no subdirectories, logs or checksums. +9. **Report** the outcome to Gatus on every exit path (§3.5). +10. **No service stop is needed.** `pg_dump` takes a consistent snapshot online. Don't + stop `arbret-server` for it. + +### 3.4 Reference script + +This is the same shape as the script every other service on the host uses (our +`backup_source` role), adapted to `pg_dump`. Adjust the database name and the +`pg_dump` invocation to match your current role. + +`/usr/local/bin/arbret-backup.sh` (`root:root 0750`): + +```bash +#!/usr/bin/env bash +set -euo pipefail +umask 077 + +BACKUP_DIR=/opt/backups/arbret +RECIPIENT="age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza" +RETENTION_DAYS=7 +NAME=arbret +DB=arbret # <- your database name +ARTIFACT="${BACKUP_DIR}/${NAME}_$(date +%Y%m%d_%H%M%S).dump.age" + +PUSH_URL="${GATUS_PUSH_BASE:-}/backup-dump_arbret/external" +PUSH_TOKEN="${GATUS_PUSH_TOKEN:-}" + +report() { # $1=true|false $2=message + [ -n "${GATUS_PUSH_BASE:-}" ] || return 0 + local enc; enc=$(printf '%s' "$2" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g') + curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" "${PUSH_URL}?success=$1&error=${enc}" || true +} +on_exit() { # the ONE exit trap: bash keeps only one, a second replaces it + local rc=$? + if [ "$rc" -ne 0 ]; then report false "arbret dump exited ${rc}" + elif [ ! -s "$ARTIFACT" ]; then report false "arbret produced no artefact" + else report true "arbret $(du -h "$ARTIFACT" | cut -f1)"; fi +} +trap on_exit EXIT + +command -v age >/dev/null || { echo "age not installed" >&2; exit 1; } +getent group backup-pull >/dev/null || { echo "group backup-pull missing - personal_infra has not provisioned this host" >&2; exit 1; } + +install -d -o root -g backup-pull -m 0750 "$BACKUP_DIR" +rm -f "${BACKUP_DIR}/${NAME}_"*.partial + +runuser -u postgres -- pg_dump -Fc "$DB" | age -r "$RECIPIENT" -o "${ARTIFACT}.partial" +chown root:backup-pull "${ARTIFACT}.partial"; chmod 0640 "${ARTIFACT}.partial" +mv "${ARTIFACT}.partial" "$ARTIFACT" + +find "$BACKUP_DIR" -maxdepth 1 -type f -name "${NAME}_*.dump.age" -mtime +"$RETENTION_DAYS" -delete +echo "wrote $ARTIFACT" +``` + +Units: + +```ini +# /etc/systemd/system/arbret-backup.service (0644; the token is in the env file, not here) +[Unit] +Description=arbret database backup +After=postgresql.service + +[Service] +Type=oneshot +EnvironmentFile=/etc/arbret/gatus.env +ExecStart=/usr/local/bin/arbret-backup.sh + +# /etc/systemd/system/arbret-backup.timer +[Unit] +Description=arbret database backup, nightly + +[Timer] +OnCalendar=*-*-* 02:00:00 +Persistent=true + +[Install] +WantedBy=timers.target +``` + +`age` and `curl` come from Debian (`apt install age curl`). Installing them from your +playbook is fine, since both repos asking for the same package doesn't conflict. + +### 3.5 Reporting the backup + +The script above already reports, on every exit path, to `backup-dump_arbret`, using the +same `/etc/arbret/gatus.env` as the analytics push. Its heartbeat is 26 hours: one +missed night alarms. The box-side check (`backup-store_arbret`) independently notices +within 6 hours of the next pull if the artefact stops arriving or stops being fresh. +So a broken dump and a broken pull show up as **different** red lights. + +### 3.6 Verify after your first deploy + +```bash +sudo systemctl start arbret-backup.service +sudo journalctl -u arbret-backup.service -n 20 --no-pager +ls -l /opt/backups/arbret/ # root backup-pull, -rw-r----- +sudo -u backup-pull head -c 30 /opt/backups/arbret/arbret_*.dump.age # must print "age-encryption.org/v1" +systemctl list-timers arbret-backup.timer +``` + +Then check that `backup-dump_arbret` is green on the status page, and tell us. We'll +trigger a pull rather than wait for 04:00, and confirm `backup-store_arbret`. + +### 3.7 Restoring + +Only the infra owner can decrypt: the age identity is offline. They'll hand you a +decrypted dump over a secure channel. The latest pre-incident one is +`arbret_20261016_020001.dump.age` (31.9 MB, sha256 `2bb771957513654cb907e5d63153e7714d54c7d5695d7caf09a05f948dcf515d`). +Anything written to arbret after 2026-10-16 02:00 UTC is not in any backup. + +For a custom-format dump: + +```bash +sudo -u postgres createdb -O arbret arbret # or let --create do it +sudo -u postgres pg_restore -d arbret --no-owner --role=arbret -j 4 arbret_20261016_020001.dump +``` + +Create the roles the app expects first. `pg_dump -Fc` doesn't include them. Then shred +the plaintext dump: `shred -u arbret_20261016_020001.dump`. + +--- + +## 4. What you'll receive from us + +| Item | Secret | Used for | +|---|---|---| +| Gatus push token, **prod** | yes | `/etc/arbret/gatus.env` on prd-vipy. Keep it in your vault. | +| Gatus push token, **staging** | yes | the same file on arbret-staging-box, if you do the staging pushes | +| age recipient `age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza` | **no**, it's a public key | the backup script. Safe to commit in plain text. | +| SSH access to prd-vipy for your playbooks | access | being arranged separately | +| The decrypted 2026-10-16 dump | yes, it's the full prod DB | restore, handed over once | + +You need **no** SSH key for the backup pull, and nothing from the old Uptime Kuma setup. + +## 5. What we need from you + +1. The local port arbret listens on (for the `arbret.com` vhost), and whether + `www.arbret.com` should be served or redirected. +2. The exact `/api/health` response when healthy, and confirmation that it returns a + non-200 when the DB is down. +3. The analytics timer's schedule (sets the heartbeat). +4. Whether port 4117 over the tailnet is still needed. +5. The final unit names, if not `arbret-server`, `arbret-worker`, `postgresql`. +6. Whether staging should be backed up, and whether staging pushes are in scope. +7. Who besides the infra owner should get Signal alerts for arbret. + +Once we have 1 to 3, we register the endpoints, the vhost and the pull source. Then you +can deploy and see green on the first run.