counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

jar-lid-archive: a staging deployment on its own box

Browse source 
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
counterweight 2026-10-03 23:56:30 +02:00
parent 97c7d580bc
commit c3a7ca0ba7
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
22 changed files with 929 additions and 215 deletions
Download patch file Download diff file Expand all files Collapse all files

5
ansible/group_vars/all/main.yml
View file

@ -62,6 +62,11 @@ subdomains:
# has nowhere to carry a second credential. Same call made for monero above.
music: music
# Jar Lid Archive staging (jar-lid-archive-staging-box, proxied via vipy).
# Public with no basic auth: the one-off photo links are for people without
# an account, and the app has its own login for everything else.
jar_lid_archive_staging: jars-staging
# Read by plays across several groups, so global rather than group_vars/<group>.
headscale_namespace: counter-net

430
ansible/group_vars/all/vault.yml
View file

@ -1,214 +1,218 @@
$ANSIBLE_VAULT;1.1;AES256
66356262646437623239383039643762363139653565356263623438303833663431396566353163
3631323563343638346464386661346235656130383238370a353236623366313561646334363639
35346232376138666430363437346638326264646562386436323962633630663136626462343762
3862333139646636650a656337306533313436333239653561333231363030633339303835396637
61633763313464353933306333383061346563623637383166646332303237653065346233623134
39386130613866386235376266643062653833643132633235393931636366376164306331303762
33393562303738393364323465653163633730633636343764646337383733643362396132303063
32336461396561663063633137613234356535383863346334666465643437613962376439356433
66353532333933343462636163353637613637356530333331636363616265376263636538656566
62643137623933363237343865613131383766633561636664616434393538646332346132336630
62333361626339626638333666393735373834396431343965386532313834353837336664306530
36303138373734343735653637663466353435306166316661353738353035366535333265626330
36396538353566316634376662383564323861303063386163663061323435656261316439336539
65323063343961646435373134303039373139333265633838313636373463316231633832306261
36623934343731343938306332356261646264613063393834336332666230626534653464363335
66356331666434623735613333303466656263313964363030316231303330303735373135653763
30323635303435656639653037366430316330333033306339366361353765656337316561366263
38613265363565373832636239396362616435303536646465326132626239353361613137626632
30353764323430656631366134373263343861376666346166363131333137376466393438346461
31613435363637306432343437653636356665373838626361306332316664343266356666353238
35303464326633326438353730393462373437313864326638323262323034346464643038386462
34613435326432663566386331333430663336656239353131633261626637613035636662346136
33626131626435316566393831333534393663326465633531373932626631613063666165613766
63636463656435386333323037656533306331646561363061613537663738326162656332393161
62313338376630643031666630663337646439623862306663303335303232306161353336643334
30386537336265383334336431653637303836376261303038653739396263666161653365356133
62636638393465363239616264316331316139363063313431323165313237646437643361653539
63393932623934313062376137626266363330623766646231353561646332353565643638356633
39656565363962653038393430363666643361343131323231663566343035643631646433306464
64343735393937333036623139303531343338373162396635613863316466393235343865333065
63353662313737353737353736316261346466366666666536653064356130326334326137383933
64306331663430313838376266613763343537613965303937393364396238663266363539313531
33616133646365396336613234636630396239316633333832653138393964616261643531663836
36666265386134383430626637643766636439393636336436366265396239303335306563653962
62323763626265373037343766386433646637633838643933633738333037633336646663316335
36373534663439306137353236343565636538636333313239373936653263366637613335663233
32306536643866656461343666396565373661353561393665643162613539633634666138626266
35613663376465626363373864626266393335396461316338653262316131643633666361333537
64366533633535313633306535393732383436333762616539646463366665623866323463336263
38313365326338386664363538633232343636303135356230393065613238386366323435633931
37353537313330396463383233616532653139376263346264623366383339306139386231636466
35383761373733386133386436316431613265306264656630353561333736613734653164333031
32643237366534643062653065333632343563633034663266393961353763663534353466383561
38373332323135353932343231353433366637333437636238623235663836393735346230623438
37616639643231396564373033636162616234366262333461343139363539316332333431636565
64396639383462386239613035303565376163633637653137373631383864353532373465626630
63636531653063306131623735393963623836313566643565386336636565633130383264316533
33343030316365373164663661306161363062646361646332393935373533626331623163323038
61316666366263313531373530366639333364336563623162323938356662626565383363303533
61323730313161633035646435346234366661386339346663363438663862636361343033306134
66396365653634373235623837666537323230616334663263633135316433313734663863333561
34326134613335346661316632353764343235326630316535306666326662613436383764646366
33616135333038333834656439383733656436663433303536313961333662366564376338326661
35393839643663333130383965663538393931363662336165636435643132353461343266363265
66346635313337346661646461386336386461346133323435643939616234666665616261313731
38383265643830646364373164656133396132666236313265346337643939613831303366393530
63333330346131656134366237323661316534323330306330616564393130353136306461666361
62376637646230623264363437313161396130616231346133346665303834623833663062663436
30643736376638303065356161366561353565343139343731313263393933313239363363636638
32663932626332626366333635616335316235363561373539623265663733373030373439373831
65333763653635613964396437333833626535643666643433393031336461366562333934666234
62383632303436633965643430363763353361613762306361613066356136383131383232313961
30626531353933336563656139316338616565303261363933306164393966343037353431653233
32636631373966323838333666313964323933306638653162646166623237343662646134383337
37383039383531656337626334386638613133343133633637343738343731326663356666333161
61323765636135306563323064346465633537613361383761393361353363396237333438306166
35386338356135633365396661356535623232613765653931303063353538643130323439366438
39363936316263333562646133646464346362343761336662353764613261396364626164313664
31383839303361323461663662393736616363646232666633323134653138306161323735386638
30333365663064366535656134323338663566616163333634353939363533356364393563363333
65623838323034336530326363383237373563326334343066323738663833353930663930373965
30363766633137396561376636636566363862353062663038653534623430643766633631356562
63623762343433626363353239376539633136333038623338303961373938396632313966666532
33633865626462636137343436386661353561623035656435623566613530623163316465383664
37346261616239653138623262623930316432633531313037353938396164646534316564613238
36643137303965316238346134323639386261633338363130323134336632636564636638653861
38346566643966386465366535393166626463313039613236393831393732613537336530393936
31613234333836333732323165653931336565316564346239336663363736323763623962383261
65626332626565636162336264303964323139303831656262373161303861393366353237393136
62303361636131646335633733373966626135396630623133623733373565306234646165656138
35383065613365363438396332306236646432663731326666613838643434393866623163316631
33633931373135336438643735646266383462386431666330363066386538386335383231653334
64323366363032386662333662613562333864356238626161366566356564343339343838373437
30363133363136663036656461343534666636343932313336336637376665393564313234373430
39356362616539366237646363646364326664653232313235653831663539353530393261346532
38653134623732313466386635313730303639613561323665356633653836393237333737323864
36323736326239636565346536646566656138666630396366353930383237333338626532663131
64366339303230663530626632623066613030663533366539313866646466373439663831363936
36366636653032343064383661383664636463343165326532653861663537633238323937363839
65333030653566306465643035633162393534323731346330333861353236393732346130313034
64303736323030643533633539623638643232306435663238393566376564313162633334613462
33363931383664633433363032643536356135626662653733303339656339373735643732626535
35316131356135393933393462336163373062653133356133393830393638313838396361336635
62323463333464353939346233333462646161633036643332313337353065376561336630383163
38306666666634393766366436653436623138363830343636323833353365333762343533633835
66363730383932303661313831663566613830393530616164323963636339393339623065316534
38373662636330336632653335316463623935653961363566396132616433633831636334306236
34393761663962623563333534383739613965353033646235653532303330343961656639333337
31643462663538313930383038343964326463363935396365343466643236653738626639616434
62343337356239316162383564666131663535313633373062353030336439396533376235386131
31653865643931323364356566316435333336316166643063613164643266313834323235353863
34623936613366303161353165386662376465656366656561613466383830326162336238656236
30646632396231653263373631346330376461303231373733646231373462623131303633616166
37366435353830303234393962663334656335356661333437363637663664323430623135346262
33303233343362336261633663336635366233313466636233333836663464633637383964313133
31363461336237373337353438396132616134303432336533653737333538383534363735316334
62303666363532383131383833383536333733306163366339653038313930646531653565386335
38313561303062336232333634343866393763636131333665396430306432373734613834316636
61306138626530616230343462613036323166656334656637616139663030623932386132393762
36363364343737613165376365323361363233616532643866303632653063343431613338373633
63303263626434326539316439323239616632653632346538336535373336623739323931623462
35656365336630323035616161656237353135373366376466326264323863336166633438663834
63646562386339363235396163646334333238323932383230643964366466323064313138323464
62636239663635336332633334656631383261306437333963363464303430666462633333666638
37633830663439396133323861303735346561376434386131363864316332623137376139376231
30303765646132306261363335366565306564373261663336613336336632616231363366316135
65303966616439643863326662656439623938373833323738646535636230663136653266653533
63326235376633333366373436333534336436366434363461393763633437616262326133623563
37613161303161356632316335616639373564306161623339353066383163303431313333306635
36326435376661343237303633316462613532313232643461313065376134393338613130336161
38326463303562666662613361656161306139653431386630303561626666613037393238313463
62623533396532326332326636346134616236653638363837643131343166333564323538653638
63383261383265353331663864643933373562396565346163396434383765343962336363353261
38353034643436626632623235396261326235323937646135326532313561396361363062333366
32343666343163636233646466613633383034653438646262666133643830333834383233343136
33646437396137613466643031323538626332613066333065643237353263636435393964356464
34383265303364663233326432313230336436373866303435336666343832383737303030306534
36383036363937393965386431343561333961663565333565333261323033616630343130613934
32383166333063623235333464343864326531333165346261643131393264346534393035393431
64636139383232383332623866323363623666326361386134353030376366323039393033653061
63383338303734323637386366646365663032636463336261643533663638343661613462373736
32663566666133656165396233643334343335333731333938376236633330366262366139326133
32653163343963333739303061643936363861313938646164316564393261663765393462613837
31646636326330663464363034376230393739343961653130303166366233396262656634313061
65646664373161356134616165383265346633376136376132643234623334353563323539333065
61623265343730326164373063383937323766363466343466316364373935626138343938323261
38323739663131346138623663393832346236356631633133313436363364663536316561623631
32393562646666343537663039363165646337653564363065353433643031353865386361613534
37633639303636313261613563353136346465326163623361323230306364396362373966336262
31353565623239343564333533656236656666316638646666376138373164663138633835366162
37633036346562663261343731366161613361656534343839303062373232636339643961666432
61396665656638333435386338383638323664313139656331336131326563373235646539633835
30393235316564656131303932386336366661373737393264613364663030363862656462646463
33303766303166336132626265653431313263363131653835336634356632666230666235383262
31653837356536666266383562633437376161653664353533656331303563626539626161653562
64333337626430616330346233613566353330383361353234376236656336653735663838343333
37643162376630373531396138643861326562323133303363363837306138373434306364356364
64623037643737663138626262326630323638376336663062393930663566316632323862653135
66363664336431373966663133353030393863313538366632643932363438613035326563323163
63343565373862316230643166336438393464323032666334653631393761303337303861666134
36663637346136623564613233343432383930616339626161623933323565623861653939326165
37383436353533613736323237666433316130613464396333343965393031333138396461376632
61373166643766376234646436653566313637326434363261333831336136306137333061323035
66303335363035666366386364303861383231366339643666313636623733616562333539643465
32353566326262326637393434656133313963646435333135613630313865373430366265666632
37316539363439663131323835313761313765653564663337346462653332613430373037333064
64323731623566326664613634626265383661346462323463303638346464643366663333336339
65626265663230346338633764633436653339373031623438633638326137613635633239343834
35373132376133363861373062386235353836633030323236343666323134326135336231613566
34326263643163633935643065306165643564663831633561353061653331326566633664376432
33353835366265346266663836326465356166626432356663366566313962663862363930336334
65323337386231376361343933353865633330613466323337306137633762653931313331313339
36663965353832386662666433383934303737656336383432633632646135383835336631373861
63646164373137316662373330333465343134633466396566333334663036663039373934333832
35343661303565646464373834646234623837363562386463383237373533316638666663613633
30313437626231343062636437363836386635333862373362643532646335353863323936663034
30316461373330623130353563643035303561303666393464636533626562633165393236663936
31313061303465633332646363643866306439383636356366623565663437313365376331656666
34333930333235306636646664326561613165633233353937666238646536323131313232626366
36623237613635343134313366346661656433643136353433393532303165326564646331333035
35353365633039353161663337306463303535643062366232336239656162653532306539333035
65373735373832326337333962643330363338656134303637313337393466366263363264343631
30656433643562393537326361323733643639396537303065623936336230346664616534323262
36313864396364353836373538333130303731383437376137383938383633373063323936613464
37613166323432363232313834326565663265336138396335313230303935396266646438323833
65306666636338373239346332613464373235613137643133626261313031326462356138356632
66363465346165306533613636383938636531656631306139666237396632646634333937623032
33613439656637383537356364336265653931613835333734383836343930313838626564333065
35363735313162616238643438316535356663396339316331633232393232316131396432653639
36303335643534376536383331323562353139613130383330636334636261326462656565643138
63326365363163383932653562363833643761653839306131343737353633313737353732633463
62326563336339383661376665353230613366646132653364396338663337326136333263363838
35626537666238383435613431666365303133373166386136353061373765313632343665306439
34356139363636623639626633623030363933303539626665383733363638303832333561653130
64393437386362333731643131663263626434666661326533386362306362653732376366626139
65626662323232636134653334366639386164653535323435613863353332366630303636663339
34333633373737333732363664353437613762383064316134393139393539306564396237343332
64636339643030353365356136366264353935643830633237316532316265316635393664326533
64343030396334326135336465336637623832363065353163316465303537643861306632336535
31366338323263313965643565656430366566393130326333383862313239356437623232613865
63383863316639346632323834633739356464313865376463323633393365653033376431373138
61363035653233373036666531303331383337353839363563386333313730363664666463333735
61383638363164336265383664376463616536383533666265383961353961303833353265366633
32646330333336613631633830633837346332653437356536363532396438326364623465383133
36656231343734643830383335623930343132386336666366623161666637623862363033373236
32666332316266623230666231663162643262653231313965323137383532633663323239396139
37656533373065623161623766333335353965373731346639613566343236363239313537636239
34626533346138613764346161383337623265353563343066303236333534316137356636396565
32616136306665346566616565346162613736333836623363346232373932366365663531653435
37623539366233366539396361386163643161653731353232313034613135373039653339346637
31623138326133366333656661393837653939343064656638616630346462363065623139323131
65363036383433663566626336353432613535363239373264356237323032323634383662613763
39646161363639326330346332353965366330346632303939623136303238323431346361656462
61623838626139636163353430636461666239343337373132643334616234616339313761353331
36643165373834373437363165303861306135383236303030626463373235393234313765343563
37353932366430666539623230373161303935633565323366363232336264643961383835333066
32623366663965316662383036306462616465353139336333643936363838313538363832363330
33313861313636373831656431383434363664663734626363646365666135376331366335653064
30386235313037363365363232316338613634643235383037393635333166346632356435373039
66663862373939346434333130383065386135306661353066343330666563386136376562316134
63346435643132316434363666343964303335663131383831373430653632663638396333393932
36383664376666393761643730306336333538376637633135616166346432636464336434323031
3966
33653731643464333331646636613630383836323134363761333038383334353230663631376666
3861376232363238613234663263646137336662666263360a633661313435616338383337396664
31316234616365326363353739323230326264623562313061303339373265663435313938326634
3464636635386664620a663330396463653963613537653230333430326137333964386530343636
62383336343031363537383362643934643839653731346364636534373430303636353431646534
66313264343338623333633964376137623833613664393137343432666264656466353830346631
62616639343265646661373062376138363232376562316134303730323431336533306565646438
62666132613930646564393162316461643431666664356436356264633839333338356561333038
65333466396165363030653137663939303838663030666235356639616362346330346437306133
61653130363136643038626562653437303939393133613733326530656163323531386436656363
37303930613063366537623435396564643431346266373363333332663632663939346131373565
32663430643034663530643431346335623636663265316431336361363233383761333130353530
30373566316336646536313938353663666436643532303433643139376232646566616430366330
36646433393037373733633562306631623164356237326532653833633031633735653463343461
65646664383531346531353133313334663130393237353436623638333731326139653363383032
63356235646366333534366666383961636361626636616237323237346234383035623365376137
34646162663761343662353631393131313562326138336565383764666333383731353730376135
64656231613630646330643133373766643330656166346134363263373966363665323964396233
64333038373565343330626633323933333239623634333436666338633539316263616438626461
30643566656534666533373365613861383231373062613831653537623532353561633736633866
61383532633430323330646137613633383830393136303332643531383466353039353564393232
39326130333336663038643766303464363664333261313137663863616466666534383831613732
35303831633232306131636137656461653361346134373935333462306232323165313265323239
36343864306130303537386164653866656632633861363736336362656161356463386265336430
38323739336364303061363437346137633436653566353464323462643838313631393235643635
65626461643263656537616638623066346636663335383861376336623932343439353231343739
64663263613862626266616362313136373630636262633036386462666632623336343431333231
31333164356538616562353335313161353339653262616439636134383063346566663261323938
36633631636265396230313033373230373635643165613239623532613436376464646635326535
39333433336361343530623434356636363433653035333934653162363135626564346161346130
65313130393639393539633534653330343935376365656162383132643635313034383538333333
64346336346231313939396463643166363165393439626431386334393563646337383361326431
34346462333732643365303230383937316462623566346238393431656433323535376361623561
66333530346638623332633832623532393035343930663434343737643032303764343138326638
33386666656162393633636437656564326336343661643137323362653437323939663033316230
61643437633665643439663566663232643162656139383236356538663139333162383665343766
31326539313163333034666233316634396434316535313264313430313432313938383534663333
33653165303864353638303833646134363362306632336133306430353165363063353735336464
31303536333964333833343566383238636233366233656531343664346331386230316365626662
30316635353262303962316364386263636531396636306534306464336536343138663933326236
38653563633663356339633335376361326132356334626163616361383134306239643365343562
34323031383638313336663937663564376362663534373464386635653164636430643266343766
62326134623638343735613562366232653137613265653836326461386539646133303638613639
31373465353533383237393933626333383839313736636132646564666639336530386537663239
34653533643039633337353535393832343439346536336232373837363734366430336539373732
66646339616366313834333562303466373962656435373530666466383339653064666532663236
64393032623234623561313766343233643232303630613165383761323435313130303361346237
62323430353333383539613536646439336466633435383134626530326531643135326264376662
37393832643662386137613230633964396338616638313764373464616665316531396462326431
36666338666536613666643032376331626132316433653036656330353834323865663865366535
36313937346136353534373631396630633432313065666639623538343332636665313236336366
35613330336136623331346636646134316265613863323730663837303032303131376435643131
39353965376565356136613533623235303031646331663166396631336434623561643464656263
64376636376337336661363133663139343935613033333266663439326239646434633131316637
38643164383136343939663332653433376362363432333265623166643831303639376637366363
37336361336461363437646234663330393830393861373162663463366533366637646530373838
32633764373164646339653931653066333833363534663234663864336465333164353231663032
36333666653633643362343334303035656539636663306333613964663837366338613565626635
34393964303866313835373936376130626665303031376637396639353734373337336534376563
30643261663235323831366537613434376566323837633136306663663331333863643037336133
63623035376233393431373361653964323833396362646661343235343332636435356365346330
39663533353631643062613839643761633866373539613536326132386336373731633239323362
31383863333330366665393666363864613632636537363436613230623030353436373061633037
36326633356533666438353034343966366136316331376431396536393261383633373131656530
36336364653962303730393338313435316162393561383161646232663430616239623964623431
38383563386266343935666463636533353361623965326232663937326462366433366534306534
66323139346238343031313762353166646235313463363435393236633337303561333930376439
37613630386636636634396437623866326639343863353134346438663033353232646566336164
63376535363235653362623630636661636162313462303664393032326538613139653039653065
38626234353034643330333861316665306665313933363234623639373765303065376632643233
39633764346132356564333236306333373664366134363037633730653732643130616665343466
62356138656331626437666465663132663562336365353834353632383062326164356161366138
66663532373464316264366661343863363265616438323132646361346135653063303031323833
33306231353231316165373137616134326163313430613963653532316331323738633639356565
38343537616330376263636165396139396365643663393637626132306564613263323362663637
64356665646532326466393733303130386230363831666265396635363630336331326662363864
39383432346265376137636639613162643135343433303435336436376161303766383232656166
32323462313432643330303039373239323939626266383232393433366232636134643636646431
37376330343461393766643664393864633265393631383566396530393364346634626535373133
61623931386433396630616463353236393630343838383865666466383632363631373136393930
35633934393963626263646137623262666335306233356439633334313765353161653566613833
32346637393530313730376564656434393737383037613664666338666435373066303338623930
63333763633533373437363666616263646564613466613830336134353933626336343036646137
32633265636134333438386330663764633432376633383930333835363362613738353464636330
38613135326635643436346565633833623630623736623231306237313662666666653532333234
62383866373939326335653461653238346336636636383961613534326336653030386534653934
39323266373664346437613362373162643532636566623563303139353430343638656130303866
63346537323431613434643865353865633331653233343839376237646337613566663130306537
39343735626236653636393465633038323465656661626561656438373835343361363735386638
39623830303864636333373534396562623630653362366166386264643438373435386438383831
34633264613230643434313037323538346661346464393362613434303135313136376131666530
36373866313739323962333061343836363661346334383033613534393666636631363262323238
64313330316464336533663764386666313732616338393162393434613437623730613038356663
39383236316664313731626330323039373937366638373461373736633239666438306437666666
66316639346432316631616261626262666238346530353563356436353332313362353431646430
64643366656466333363313135653331303764633839316239333630353665666465393263633333
31306663326366643032653762646464396530643662303636393432373966393934396634343032
31346365633933313436643937626634656535323431616231393866663334303632613237633666
65396262363665353837366166633435326133356330383066613537653736656565306434303663
38313837303965633236366533386661366464613262326162316365626534356237663035396264
33356165623232323664363731383962616465653230373234323961646236356132383031346465
61396136636137303933346462356339393236313361316465346631666432626434636366396531
35313264323766663436666430313864303065313633396631656438666564313038386161313433
66383734373435633838666463643636376366633334626130333730366639663961666630383234
35343964363931663431373162653333613563366462333037363838623662303837626237396132
36366162313766366237306337333338623937363165306433616530326435363063333938393939
61373433666233393962633263386534333735303564326435623363663334313163326239653966
30656132353938326530306663613637373838323432323762306263623136653937383465613038
66346661646231623161373233653337353162636334376137373138613864353036663964336362
64303362306239633339313666303636626264353833333333613132316261333830616232333662
64623061633466336336623734616632303265363630356665303663636334656136636339353534
65636564636462346133666438623461653630333833643230623632333266663066376461316661
37623637356333386638343937613361343265663033326337666138393233646639656232333864
37343733646432353564316237363964656532386561666538633737323739383766316337386137
65656231336463623965653232323162363866373732656534376430356432653539393939343661
30626130336436343033386637393334363237373737613761363633316533623033343731356338
38363231303637323861376662356564323032316130356461653466336165363363383637313862
35666338383164666135633734386432353137363464376538663465353330376237363333653566
32376462383732656135333861323465396664353032393432373861353836376239346338393230
39623532336232343761303235393963333065653935616230656236646330323465303263663236
39336233303030656161343961353566626462616333643235623032623338646163346564383034
63333439653936653731653337396231396562383565363837643138373864613338386164346262
31613133303833323238343665353238323335623162353362646337336335323639363730386462
61633966313435313165643537626330663864623762633330383730613131396463313231343837
61623965393836303230626463363438343239363562643334653733343865313334383233343932
33313930666136613766376435636432303466626565326434636561373565306532376239643735
63613037616534656265663536373164643866653466326361313965383635626331636463383363
35356533393461376132626431383839393132663161373831366130663964313839613566643165
63353139316333666337636635303236643034623830353462626535623064653065316135343564
64363963383230313765643262396330663938366631623834376532643937316236623865613665
34653331333336333364396330646632326266626638316566363539313532323762383034323036
38343838616264333737376432366535643339323734316563386439313339653763343139306565
61383538613761353733653933316634316138313565393932623638636135303962366562623231
38346365616434663362306138373634613334613037613961383032326661356163626238666233
31653938633938366632326439333237663539383461646236306265326536626135343161396137
34616636663639396534363431666464306538623632623133623133643864313839653865623266
35643865366465346532386130353565633333613061396139383265663734323139663532376638
38623966306639363938643564343330636363346261333166613938623936303966613037663365
62376334613363323563363332356463376532323964333338623239326663343031373635663032
33626432356662636130363761323862616131653630396164623435333061366564626236643936
65316562633036366233303538373530343932616533346637656464346166306438396435666533
37313766366636336535363536333061373264663163303162666330633061356464353164636239
39623836663066623431343835623264373239363766323338353633336338363536643462666236
35313031633561353066306336336362643863393730613938373539363831336266353266633036
30313039336631396664383434316462643366643532353638396339353430303063303734343830
31323761643235663061336237333866323139613533366463353433613832316232323938323331
39356261636231353936313033363962353232663831316634653634323965313035663537323864
33633138383431376131616462363337313035326339646661643730666337393266316562333138
39616636323630356631653834313466393533333761666534333239306666356166363433326434
33643566323438333934363236313633666465663236353034653439323338613433333538336535
30326435383362343065356463373761333565396131336562636236353863613635636436326131
37353238656337633135643430306532393530653032663437303139323463613136656563613633
39303463656631336231643733323439376331653732376639303231333533373264623034316435
64386635636565333961663936356131643536636235666433356338646531306535653366363362
64336634396666303364663033383034616639646532333238336138343032376631326139646664
37323432613664343032313838663561373735323938666339336133653131383263366665656537
61363139356666303438316434626536323362306539613930353263663263353933353664393666
37663763616635306665386163333034643465376130313861633435366666653535383234363438
34326565373461343035616162316532333739386562383466626436366434306261363931653464
63306238663038373631363834313261323337393263323362306636663765393535303433643562
63616336383563343732653034316133343662333733323231623064386561623535376235336432
38646537303136653939346335653031636266343132626262643439363564613132356634666435
34626562336630363363393261643735353263653462353336383630366335643966396434363034
65623233346335343637633564393466616365626361613134306362663333353730336264616664
33656361323631663639303033663632663262313461663133646663633833383833636661336463
37633266616639663830343333336362653838316363303637333535663136336633393339666263
37313434303638616530336366303865333038326534656135613136613930376635363062353962
38323339356235663965383633323665313932643632363132386666363239633666313566653063
64643433363234396563356435383335623133613364313834396236376339633738326235363737
31303566613461386434646233306261393835613739353831306164343164616432656466636139
62393132636465663965633432353665643535643233313738323236383166653838663338323462
61383035636632323863333361643333363238636436616562393465326230373164366261313564
39343861333562373762623437613334396634613462373132386564616462333638373161653737
38313366373864333064646133393561396537353333626462666361326532323239376431326330
30373734346566663235616235316633636263373431326166623734303734383337643562666536
61353130646537623936646639393439306639356130373431623431643834386632616165383133
61666165396634613763306561356533663263386639643766306166303162376366363530396361
65356538663638633261636332386262343236663731313932656339323239343938666662343064
66353234346530366163656437356464376136663261343938636566333865313261396164316137
38663238656365383131363239333537646333656136656465636464656530306137363436636231
39366339663630653461383461323234363936346230643266393833356432396661356263333238
38656530306133363733336335386238626132356237313334633634393634333937643631623963
36326364306664613832363734336463303635373262653632356263336665353739373231316664
34633036383439393339363364346165376261613339356236323330306432616665343265363564
66666137346231383465353666633835393737313261643935656530356162613538363864343830
35323865613563366130323132326238346338303761353433656532643131333962373066336233
34303734396235303137616362653037333138306132393537646531616635636561343831366531
36333932323234643237346266323466303736663665363362373465333139393934623939313535
34316433323136316533313663393238663530386565376131366631393031643362346239316430
64303636343765326465616464663234373137393137666635306261653466323733303962393461
61356438316631333231636465653933343264356230313937636431336261306464316532343335
32383732363866383531373136353331623566636332643332636237653737313064393566373837
37333031643034343637333832363931343938303764313836373230353931613535313765393065
30623234653039316461626635366337633265623236333963323934313065623732353362303338
64663130383030613536383739646562636662656234363036613162666461373834333530363363
62383166353333663232363737653639316235303535616536383962653039386464303435356538
61653166633162616563336663323830383732666264393262323139303936626464373132303738
30343061313535363838653930306164303438353736393862643461363333396539326633386236
63623666343330363166373037636332313966303565303061663361386239353962666164383435
64626663383735393431623436323862656262396461393663616530353132646234353838633537
66313437306330653963613663373235383338383439636563643333323265626132326639386639
38663762326666343935393664643761393236323266336265393033643734333439353538613062
38626438663431366464343937316165313937323732643061643334643437353764333231646362
65393566663530383133393430383237633638363363313963383230663434353635353364333830
30306535353962633865346131333035353638336539336635643238383762303563323663616164
63356630393664653637613935363536656239616239316337353730666632633234613331336136
34383036616432306533663164316163303332313239646662656536333564643539343533383939
32303935313565626661393334303564633136323164623965663364323966346636396532336535
38373232616265346131633930333134666333663961383063666237653462623635323537393837
30373438616561353430633139323163353138363735636530643630346566633631383931363638
66633266643762626530613238343663326236363134316337653861376561383631663937373833
32663063356164333731343263393837386332623665313135396665373436323366613331363333
65393533353933643134643362366330653261313064663963336539653134396437383430313666
33663234616461626630646563616561313139396630646461386338623266666434376335306435
36643430663131373064313464633733356131386338643439653032353436656661356366626533
35343434396361303634636438633036386563383664646561663566613339323865363136333230
37616265613030633061376639386461383062353135666534346632353064353231

28
ansible/host_vars/jar_lid_staging_local/main.yml Normal file
View file

@ -0,0 +1,28 @@
---
# Jar Lid Archive STAGING. Secrets are in vault.yml beside this file.
#
# These live here rather than in the role defaults because three plays read
# them: the deploy on this box, the Caddy vhost on the edge host (port, domain)
# and the backup (retention). A role default cannot reach a second play.
# Names this deployment everywhere it must be unique across hosts: the Caddy site
# file, the Gatus probe (probe_<instance>), the backup artefacts and the
# backup-dump/backup-store endpoints.
jar_lid_archive_instance: jar-lid-archive-staging
# Fixed by the image's CMD (gunicorn --bind 0.0.0.0:8000); changing this moves
# what Caddy and the probe call, not where gunicorn listens.
jar_lid_archive_port: 8000
jar_lid_archive_domain: "{{ subdomains.jar_lid_archive_staging }}.{{ root_domain }}"
# Staging data is disposable: three days locally and on small-backups-box.
jar_lid_archive_backup_retention_days: 3
jar_lid_archive_seed_users:
- {username: alice, email: alice@example.com, password: "{{ jar_lid_archive_seed_password_alice }}"}
- {username: bob, email: bob@example.com, password: "{{ jar_lid_archive_seed_password_bob }}"}
- {username: admin, email: admin@example.com, password: "{{ jar_lid_archive_seed_password_admin }}", is_superuser: true}
# No systemd units of ours run here - the stack is docker compose - so infra/401
# has nothing to watch. The probe in the deploy playbook calls /health/, which
# checks the database and the bucket as well as gunicorn.

34
ansible/host_vars/jar_lid_staging_local/vault.yml Normal file
View file

@ -0,0 +1,34 @@
$ANSIBLE_VAULT;1.1;AES256
37383535306133363031316535656461613038653562373839363432623163356234303061643566
6533333535613064653336313664643063326562666639640a373530386539316363346332663137
31303230323065623733663666383039346337343239646465393166663239643836356639373765
3564313065363266360a643239333330616463616533623435656164386264303634356134303831
36356131323637363734316161383065653663623732386638363165633265356362386538643039
64636464323730316638316430643339356662353736366332663738656566663137366166393735
34633430363931323330393234336661383532653366633735313365363562396137356438353330
65313737313131326361326631363864396536343633623963303632623465363964373964393430
61323161663939346435386435616364653336646165356232346335353235356339656336306138
65613636353636623562613362633761333637613066663838396638613161663532393030303035
35316464356539666631636265646433353537396564366538643962373364383730626164316265
63393034663835346362636564316135326430633963393139646365333666663630336462323361
31353830633335366635366435643864313665366563396134313938666537663232343162383366
34636231636261663739383366363962366438323962393732393864623238343961363035333866
61386630396334626564306536336561333265383464623730386263373132613438396332356631
30386435333739653637633032313966626437303634613838653735373666343565306663376230
61303734623865353835626464613033346437623234626136353664303434386262346634356463
34396433353431393935386136333331643539633138636135353339323930386634393639666665
36643462653061363062383039313936303062393364353366623965313530303031306336396336
65643663343163636338393666386134366337633064376530303463393737396165663930383562
64316363663636383764656139366433393766613262613433623637393538633130616232373136
31376636633666626365376533356262393065656539613266326465643663383439373661393631
30636136363066653936653837376634356333353036633165316662383337636435323162316430
30316331353361396132653233623034376539323064363135306262376638353664643163653334
37643638323038663531376336313064383238366430663363353733356530393661646138303230
30633862626166653362346632626261636531373130353034656532386230616330643134383631
63396138326336626431656130306531626539323934386332306361396239376466623866613364
36666632316331326361373136326465323365653034363337313135656262623465306439646362
30636639663163643430363165316231366232623332393635623461306339313762323265623732
30633464343235323566356434376535313238663533633630356163336532303765633366316166
38633462303762323930393337306265393033653061616337646163333335333737666332666632
35613132646636333335313263343136323337616662623465653939336637653134383766336265
35633638663636353537333238643236623033636163333164663062353863643164

5
ansible/infra/402_public_monitoring.yml
View file

@ -37,6 +37,7 @@
- {sub: "{{ subdomains.monero }}", host: vipy, interval: '24h'}
- {sub: "{{ subdomains.monero_explorer }}", host: vipy, interval: '24h'}
- {sub: "{{ subdomains.music }}", host: vipy}
- {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy}
# A public resolver on purpose: this must test what the internet sees, not
# what a local cache or the tailnet's MagicDNS happens to answer.
@ -75,6 +76,10 @@
# API the mobile clients speak cannot carry a proxy credential, so its
# own login is the gate. See roles/navidrome/README.md.
- {name: music, sub: "{{ subdomains.music }}", path: "/app/", status: 200}
# /health/ through the whole public path: DNS, vipy's Caddy and cert, the
# tailnet hop, gunicorn, and the database and bucket behind it (503 if
# either is down). "/" would only prove the login redirect works.
- {name: jars-staging, sub: "{{ subdomains.jar_lid_archive_staging }}", path: "/health/", status: 200}
# Ports published from the edge host by socket_proxy.
public_tcp:

1
ansible/infra/403_service_probe_registration.yml
View file

@ -55,6 +55,7 @@
- {name: monerod, host: monerod_box_local, heartbeat: '26h'}
- {name: xmrblocks, host: monerod_box_local, heartbeat: '26h'}
- {name: navidrome, host: music_box_local}
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
tasks:
- name: Build the probe endpoint list

8
ansible/inventory.ini
View file

@ -21,6 +21,10 @@ monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=cou
# key was still in known_hosts and SSH refused the connection, which is the
# designed behaviour described in group_vars/nodito_vms.yml.
music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# lan_ip read from the Proxmox guest agent for VM 3300 (qm guest cmd
# network-get-interfaces), and the host key checked against the one the agent
# reads off the disk before it went into known_hosts.
jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1.151 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
[lapy]
@ -63,6 +67,10 @@ memos_box_local
[music]
music_box_local
# Staging only for now; a production host joins this group with its own host_vars.
[jar_lid_archive]
jar_lid_staging_local
[ci_runner]
forgejo_runner_local

7
ansible/playbooks/backups.yml
View file

@ -30,6 +30,10 @@
- name: forgejo
source: "backup-pull@prd-vipy:/opt/backups/forgejo/"
retention_days: 14
# Staging: disposable data, so the long tail is as short as the local one.
- name: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}"
source: "backup-pull@{{ hostvars['jar_lid_staging_local'].ansible_host }}:/opt/backups/{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}/"
retention_days: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_backup_retention_days }}"
# ─────────────────────────────────────────────────────────────────────────────
# Register the backup checks with Gatus.
@ -62,7 +66,8 @@
- {name: vaultwarden, host: vipy}
- {name: lnbits, host: vipy}
- {name: forgejo, host: vipy}
store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo]
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo, jar-lid-archive-staging]
tasks:
# 26h, not 7h: the DUMP is genuinely daily, so the window cannot be tighter

121
ansible/roles/jar_lid_archive/README.md Normal file
View file

@ -0,0 +1,121 @@
# `jar_lid_archive`
Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive)
as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and
SeaweedFS as the S3 photo store (`storage`). The same three services as the app
repo's own `compose.yaml`, so staging rehearses what dev and CI run.
Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to
every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local`
(jar-lid-archive-staging-box, nodito VM 3300), public at
`jars-staging.contrapeso.xyz` through vipy's Caddy.
Per-host values are in `host_vars/<host>/main.yml`, and secrets are in `vault.yml`
beside it. A production host would be a second inventory entry with its own
`jar_lid_archive_instance`, domain and vault, not a second playbook.
## Updates: follows `:master`
CI pushes `:master` on every green build of master. Every run of the playbook
runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only
if the digest moved. There is no watchtower, so **a merge reaches staging when
the playbook runs**, not on its own. The run ends by printing `/health/`, which
includes the deployed version.
`up --wait` only returns once every container's healthcheck passes. `web`'s is
`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means
the app actually serves.
## Networking: why `web` is on the host network
Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT`
rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well
as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids
that, but at boot Docker races tailscaled for the address and loses.
So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host
port:
- **LAN:** UFW default-deny blocks it.
- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW
(checked on mempool-box), so the headscale ACL is the real control. It grants
`tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`.
- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches
them from the host network.
This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto`
(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only
thing that can reach gunicorn. Caddy overwrites whatever value the client sent.
## Secrets and seed users
The Django secret key, the Postgres password, the S3 secret and the three seed
passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They
reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose
interpolates. `docker-compose.yml` itself has no secrets.
The seed users are alice, bob and admin (superuser), with the fixture's usernames
and **vault** passwords. The fixture's `*-test-pass` passwords are published in
the app repo and this site is public. `seed_users` runs on every deploy and
resets each listed user's password, so rotating one means editing the vault and
re-running the playbook.
```sh
ansible-vault view host_vars/jar_lid_staging_local/vault.yml # from ansible/
```
## Backups
`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses
`backup_source`. At 02:20 it stops `web` only, and
`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with:
- `db.dump`: `pg_dump -Fc`
- `seaweedfs/`: the photo store's data directory, as files
`web` is the only writer to the database and the photo store. With it stopped,
both are quiet and are captured at the same instant. `storage` is deliberately
left running: on this disk it can take minutes to restart, and the first test run
that stopped it left the app down for 26 minutes. Uploads fail while `web` is
down.
The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally,
and small-backups-box keeps three as well. Staging data is disposable.
### Restore
```sh
age -d -i ~/.age/counterweight_age jar-lid-archive-staging_<ts>.tar.age | tar -xf - -C /tmp/restore
# on the box, stack down:
docker compose -f /opt/jar-lid-archive/docker-compose.yml down
rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db
docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \
pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait
```
## Monitoring
| What | Where |
|---|---|
| Host liveness and disk | infra/400 (automatic for `managed`) |
| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) |
| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) |
| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` |
infra/401 has nothing to watch here, because no systemd units of ours run on
this box.
## Size and disk
The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's
random reads keep that mirror close to saturated, and this box has measured
**~96% "full" I/O pressure** with nothing of its own running. Everything is slow
here, but especially container starts. The healthcheck `start_period`s are sized
for that. Moving the disk to `local-zfs` (NVMe) would fix it.
1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian,
the images, the photos and three days of local backups, and each backup is
about the size of the whole photo store. infra/400's disk check alarms at 85%.

57
ansible/roles/jar_lid_archive/defaults/main.yml Normal file
View file

@ -0,0 +1,57 @@
---
# Jar Lid Archive: Django (gunicorn) + Postgres + SeaweedFS (S3) as one compose
# stack. Source and CI live in forgejo.contrapeso.xyz/counterweight/jar-lid-archive.
#
# Per-deployment values (domain, port, retention) live in the host's host_vars,
# because the Caddy play on the edge host and the backup plays read them too and
# a role default cannot reach a second play.
jar_lid_archive_dir: /opt/jar-lid-archive
# ── Images ───────────────────────────────────────────────────────────────────
# The app follows a moving tag on purpose: every run pulls it, and compose
# recreates `web` only when the digest actually changed. CI pushes :master on
# every green build of master, plus :<sha>.
jar_lid_archive_image: forgejo.contrapeso.xyz/counterweight/jar-lid-archive
jar_lid_archive_image_tag: master
# Pinned to what the app's own compose.yaml and CI run, so staging rehearses the
# same stack. Bump them together with the app repo, not separately.
jar_lid_archive_postgres_image: "postgres:18"
jar_lid_archive_seaweedfs_image: "chrislusf/seaweedfs:4.47"
# Registry login. Empty = pull anonymously, which works today: the package is
# owner-scoped in Forgejo and public even though the repository is private.
jar_lid_archive_registry: forgejo.contrapeso.xyz
jar_lid_archive_registry_user: ""
jar_lid_archive_registry_token: ""
# ── Networking ───────────────────────────────────────────────────────────────
# gunicorn listens on the HOST network on this port. See the README for why host
# networking rather than a published port: Docker's published ports bypass UFW.
jar_lid_archive_port: 8000
# Loopback-only ports for the two backing services, so the web container (on the
# host network) can reach them and nothing else can.
jar_lid_archive_postgres_port: 5432
jar_lid_archive_s3_port: 8333
jar_lid_archive_domain: "" # e.g. jars-staging.contrapeso.xyz
# ── App settings ─────────────────────────────────────────────────────────────
jar_lid_archive_postgres_db: jarlid
jar_lid_archive_postgres_user: jarlid
jar_lid_archive_s3_access_key: jarlid
jar_lid_archive_s3_bucket: photos
# Secrets: pass from the vault. The assert in tasks/deploy.yml refuses empties.
jar_lid_archive_django_secret_key: ""
jar_lid_archive_postgres_password: ""
jar_lid_archive_s3_secret_key: ""
# ── Seed users ───────────────────────────────────────────────────────────────
# Loaded with `manage.py seed_users -` on every run. That command RESETS each
# listed user's password and flags, so the vault stays the source of truth.
# Same usernames as the app's tests/fixtures/users.json, never its passwords:
# this deployment is reachable from the internet.
# - {username: alice, password: "...", email: "...", is_superuser: false}
jar_lid_archive_seed_users: []

100
ansible/roles/jar_lid_archive/tasks/deploy.yml Normal file
View file

@ -0,0 +1,100 @@
---
- name: Assert the deployment is fully specified
ansible.builtin.assert:
that:
- jar_lid_archive_domain | length > 0
- jar_lid_archive_django_secret_key | length >= 50
- jar_lid_archive_postgres_password | length > 0
- jar_lid_archive_s3_secret_key | length > 0
fail_msg: >-
jar_lid_archive needs a domain and its three secrets from the vault. The
app falls back to development defaults for every one of them, so an empty
value here would not fail - it would deploy with a published secret key.
no_log: true
- name: Create the deployment directory
ansible.builtin.file:
path: "{{ jar_lid_archive_dir }}"
state: directory
owner: root
group: root
mode: '0755'
# Existence only, no owner: the postgres and seaweedfs entrypoints chown their
# own data directories to the uid inside the container. Declaring an owner here
# would fight them on every run - the mempool MariaDB lesson.
- name: Ensure the data directories exist
ansible.builtin.file:
path: "{{ jar_lid_archive_dir }}/{{ item }}"
state: directory
loop:
- postgres
- seaweedfs
# diff: false, because --diff prints rendered content and this is all secrets.
- name: Write the compose environment file
ansible.builtin.template:
src: env.j2
dest: "{{ jar_lid_archive_dir }}/.env"
owner: root
group: root
mode: '0600'
diff: false
- name: Write docker-compose.yml
ansible.builtin.template:
src: docker-compose.yml.j2
dest: "{{ jar_lid_archive_dir }}/docker-compose.yml"
owner: root
group: root
mode: '0644'
- name: Log in to the container registry
ansible.builtin.command:
cmd: "docker login {{ jar_lid_archive_registry }} -u {{ jar_lid_archive_registry_user }} --password-stdin"
stdin: "{{ jar_lid_archive_registry_token }}"
when: jar_lid_archive_registry_token | length > 0
changed_when: false
no_log: true
# Pulled on every run: staging follows :master. `up` below recreates `web` only
# when the pulled digest differs from the running one.
- name: Pull images
ansible.builtin.command: docker compose pull --quiet
args:
chdir: "{{ jar_lid_archive_dir }}"
changed_when: false
# --wait blocks until every service's healthcheck passes, and `web`'s is
# /health/, which itself checks the database and the bucket - so this task
# succeeding means the app actually serves, not merely that containers exist.
- name: Start the stack and wait for it to be healthy
ansible.builtin.command: docker compose up -d --remove-orphans --wait --wait-timeout 300
args:
chdir: "{{ jar_lid_archive_dir }}"
register: jar_lid_archive_up
changed_when: jar_lid_archive_up.stderr is search('Created|Recreated|Started|Removed')
- name: Read the deployed version
ansible.builtin.uri:
url: "http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
return_content: true
register: jar_lid_archive_health
# Under --check the stack was never started, so there is nothing to ask.
when: not ansible_check_mode
- name: Show the deployed version
ansible.builtin.debug:
msg: "{{ jar_lid_archive_health.json }}"
when: not ansible_check_mode
# Used by the backup (services/jar-lid-archive/setup_backup_jar_lid_archive.yml).
# Lives here because what to dump is a fact about this stack's layout.
- name: Install the dump script
ansible.builtin.template:
src: dump.sh.j2
dest: /usr/local/bin/jar-lid-archive-dump.sh
owner: root
group: root
mode: '0755'
validate: "bash -n %s"

74
ansible/roles/jar_lid_archive/tasks/docker.yml Normal file
View file

@ -0,0 +1,74 @@
---
# Same Docker CE install as roles/mempool/tasks/docker.yml. infra/910_docker
# targets every managed host, so each Docker role installs its own engine.
- name: Remove old Docker-related packages
apt:
name:
- docker.io
- docker-doc
- docker-compose
- podman-docker
- containerd
- runc
state: absent
purge: yes
autoremove: yes
- name: Update apt cache
apt:
update_cache: yes
- name: Install prerequisites
apt:
name:
- ca-certificates
- curl
state: present
- name: Create directory for Docker GPG key
file:
path: /etc/apt/keyrings
state: directory
mode: '0755'
- name: Download Docker GPG key
get_url:
url: https://download.docker.com/linux/debian/gpg
dest: /etc/apt/keyrings/docker.asc
mode: '0644'
- name: Get Debian architecture
command: dpkg --print-architecture
register: deb_arch
changed_when: false
- name: Add Docker repository
apt_repository:
repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable"
filename: docker
state: present
update_cache: yes
- name: Install Docker packages
apt:
name:
- docker-ce
- docker-ce-cli
- containerd.io
- docker-buildx-plugin
- docker-compose-plugin
state: present
update_cache: yes
- name: Ensure Docker is started and enabled
systemd:
name: docker
enabled: yes
state: started
- name: Add user to docker group
user:
name: "{{ ansible_user }}"
groups: docker
append: yes

6
ansible/roles/jar_lid_archive/tasks/main.yml Normal file
View file

@ -0,0 +1,6 @@
---
# import_tasks, not include_tasks: static imports stay visible to --list-tasks,
# which is the main verification tool in this repo.
- ansible.builtin.import_tasks: docker.yml
- ansible.builtin.import_tasks: deploy.yml
- ansible.builtin.import_tasks: seed.yml

13
ansible/roles/jar_lid_archive/tasks/seed.yml Normal file
View file

@ -0,0 +1,13 @@
---
# seed_users is idempotent but RESETS each listed user's password and flags, so
# re-running it is how a rotated vault password reaches the app. It reports
# "N created, M updated"; only a creation counts as a change.
- name: Seed users
ansible.builtin.command:
cmd: docker compose exec -T web python manage.py seed_users -
chdir: "{{ jar_lid_archive_dir }}"
stdin: "{{ jar_lid_archive_seed_users | to_json }}"
register: jar_lid_archive_seed
changed_when: "jar_lid_archive_seed.stdout is not search('Seeded users: 0 created')"
when: jar_lid_archive_seed_users | length > 0
no_log: true

123
ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2 Normal file
View file

@ -0,0 +1,123 @@
# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
#
# Same three services as the app repo's compose.yaml, with three differences:
# - `web` runs the CI-built image instead of `build: .`
# - `web` is on the HOST network, so UFW and the tailnet ACL decide who reaches
# gunicorn. A published port would be DNATed by Docker straight past UFW and
# answer on the LAN too. See the role README.
# - data is bind-mounted under {{ jar_lid_archive_dir }} rather than in named
# volumes, so the backup tars a path it can name.
#
# Secrets are interpolated from .env (0600) beside this file; this file has none.
name: jar-lid-archive
x-logging: &logging
# Docker's default json-file driver never rotates, on a 10 GB disk.
driver: json-file
options:
max-size: "10m"
max-file: "3"
x-postgres-env: &postgres-env
POSTGRES_DB: "{{ jar_lid_archive_postgres_db }}"
POSTGRES_USER: "{{ jar_lid_archive_postgres_user }}"
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?}
x-s3-env: &s3-env
S3_ACCESS_KEY: "{{ jar_lid_archive_s3_access_key }}"
S3_SECRET_KEY: ${S3_SECRET_KEY:?}
services:
web:
image: "{{ jar_lid_archive_image }}:{{ jar_lid_archive_image_tag }}"
# gunicorn binds 0.0.0.0:{{ jar_lid_archive_port }} - the port is fixed by the image's CMD.
network_mode: host
environment:
<<: [*postgres-env, *s3-env]
POSTGRES_HOST: 127.0.0.1
POSTGRES_PORT: "{{ jar_lid_archive_postgres_port }}"
S3_ENDPOINT_URL: "http://127.0.0.1:{{ jar_lid_archive_s3_port }}"
S3_BUCKET: "{{ jar_lid_archive_s3_bucket }}"
DJANGO_DEBUG: "0"
DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY:?}
# 127.0.0.1 for the container healthcheck and the probe, which both call
# /health/ directly rather than through Caddy.
DJANGO_ALLOWED_HOSTS: "{{ jar_lid_archive_domain }},127.0.0.1,localhost"
# Caddy terminates TLS and forwards plain HTTP with X-Forwarded-Proto. The
# app must trust that header or every POST fails CSRF and share links come
# out http://. Safe only because nothing but Caddy can reach gunicorn.
DJANGO_BEHIND_TLS_PROXY: "1"
DJANGO_CSRF_TRUSTED_ORIGINS: "https://{{ jar_lid_archive_domain }}"
depends_on:
db:
condition: service_healthy
storage:
condition: service_healthy
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:{{ jar_lid_archive_port }}/health/')"]
interval: 30s
timeout: 5s
retries: 5
start_period: 180s
logging: *logging
restart: unless-stopped
db:
image: "{{ jar_lid_archive_postgres_image }}"
environment: *postgres-env
ports:
- "127.0.0.1:{{ jar_lid_archive_postgres_port }}:5432"
volumes:
- ./postgres:/var/lib/postgresql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 3s
retries: 15
start_period: 120s
logging: *logging
restart: unless-stopped
storage:
image: "{{ jar_lid_archive_seaweedfs_image }}"
environment: *s3-env
# Copied from the app's compose.yaml: write the S3 identity from the
# environment, then hand over to the image's entrypoint, which drops root
# and runs master + volume + filer + S3 gateway.
#
# The `rm -f` is ours, and load-bearing. On a RESTART of the same container
# (a reboot, a backup's stop/start) /tmp/s3.json survives, owned by seaweed
# from the chown below. Root cannot then re-create it: /tmp is sticky and
# the kernel's protected_regular refuses O_CREAT on another user's file
# there. The result was "can't create /tmp/s3.json: Permission denied", an
# unhealthy storage container, and web never starting - found on the first
# backup run.
entrypoint: ["sh", "-c"]
command:
- |
rm -f /tmp/s3.json
umask 077
cat > /tmp/s3.json <<EOF
{"identities": [{"name": "app",
"credentials": [{"accessKey": "$$S3_ACCESS_KEY", "secretKey": "$$S3_SECRET_KEY"}],
"actions": ["Admin", "Read", "Write", "List", "Tagging"]}]}
EOF
chown seaweed /tmp/s3.json
exec /entrypoint.sh server -s3 -s3.config=/tmp/s3.json
ports:
- "127.0.0.1:{{ jar_lid_archive_s3_port }}:8333"
volumes:
- ./seaweedfs:/data
# start_period, because this VM's disk is on nodito's HDD mirror, shared
# with knots-box. Under that contention SeaweedFS has taken over the 150s
# that retries x interval allowed; compose then marked it unhealthy, and
# web's depends_on gave up and never started it - a 26-minute outage.
# Failures inside start_period do not count.
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8333/healthz"]
interval: 5s
timeout: 3s
retries: 30
start_period: 300s
logging: *logging
restart: unless-stopped

25
ansible/roles/jar_lid_archive/templates/dump.sh.j2 Normal file
View file

@ -0,0 +1,25 @@
#!/usr/bin/env bash
# Jar Lid Archive dump - managed by Ansible (roles/jar_lid_archive).
#
# Writes ONE uncompressed tar to STDOUT, for roles/backup_source to encrypt:
# db.dump pg_dump custom format -> pg_restore -d jarlid db.dump
# seaweedfs/ the photo store, as files
#
# Called with `web` stopped (the backup playbook arranges it). web is the only
# writer to db and storage, so both are quiescent and agree with each other: no
# row points at a photo the tar does not have. storage keeps running; with no
# writes arriving its volume files do not change under tar.
#
# The pg_dump lands in a temp dir on the same disk as the live database, which
# is already plaintext; it is gone before this script exits.
#
# tar, not tar.gz: the bulk is JPEGs, which do not compress, and gzip on one
# core would only make the window longer.
set -euo pipefail
cd {{ jar_lid_archive_dir }}
tmp=$(mktemp -d {{ jar_lid_archive_dir }}/.dump.XXXXXX)
trap 'rm -rf "$tmp"' EXIT
docker compose exec -T db pg_dump -U {{ jar_lid_archive_postgres_user }} -d {{ jar_lid_archive_postgres_db }} -Fc > "$tmp/db.dump"
tar -cf - -C "$tmp" db.dump -C {{ jar_lid_archive_dir }} seaweedfs

5
ansible/roles/jar_lid_archive/templates/env.j2 Normal file
View file

@ -0,0 +1,5 @@
# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
# Read by docker compose to interpolate docker-compose.yml. 0600: secrets.
POSTGRES_PASSWORD={{ jar_lid_archive_postgres_password }}
S3_SECRET_KEY={{ jar_lid_archive_s3_secret_key }}
DJANGO_SECRET_KEY={{ jar_lid_archive_django_secret_key }}

7
ansible/services/headscale/templates/acl.json.j2
View file

@ -43,6 +43,7 @@
"tag:mempool-box:8080",
"tag:memos-box:5230",
"tag:music-box:4533",
"tag:jar-lid-archive-staging-box:8000",
"tag:monerod-box:18089",
"tag:monerod-box:8081",
"tag:arbret-staging-box:80",
@ -82,7 +83,8 @@
"tag:prd-arbret:22",
"tag:prd-vipy:22",
"tag:memos-box:22",
"tag:stalwart-box:22"
"tag:stalwart-box:22",
"tag:jar-lid-archive-staging-box:22"
],
"src": [
"tag:small-backups-box"
@ -102,6 +104,9 @@
"tag:fulcrum-box": [
"{{ headscale_namespace }}@"
],
"tag:jar-lid-archive-staging-box": [
"{{ headscale_namespace }}@"
],
"tag:knots-box": [
"{{ headscale_namespace }}@"
],

51
ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml Normal file
View file

@ -0,0 +1,51 @@
---
# Jar Lid Archive on its own box, published from the edge host.
#
# See roles/jar_lid_archive/README.md. Per-host values (domain, port, seed users,
# secrets) are in host_vars/<host>/, so a production host is a new inventory
# entry in [jar_lid_archive], not a new playbook.
- name: Deploy Jar Lid Archive
hosts: jar_lid_archive
become: yes
roles:
- jar_lid_archive
# "Does the app answer", as opposed to "are the containers up". /health/
# runs SELECT 1 and touches the bucket, and returns 503 if either fails.
- role: healthcheck
vars:
healthcheck_name: jar-lid-archive
healthcheck_description: "Jar Lid Archive /health/ (app, database and photo store)"
healthcheck_command: "curl -fsS --max-time 10 -o /dev/null http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_{{ jar_lid_archive_instance }}/external"
healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
- name: Publish Jar Lid Archive from the edge host
hosts: edge
become: yes
tasks:
# No basic auth: the one-off links (/links/...) are opened by people who have
# no account and no proxy credential. The app's own login guards the rest.
#
# Caddy sets X-Forwarded-Proto: https and overwrites any value the client
# sent, which is what lets the app trust it (DJANGO_BEHIND_TLS_PROXY).
#
# No request size limit is set, deliberately: Caddy has none by default, and
# phones upload multi-megabyte photos.
- name: Expose Jar Lid Archive over HTTPS
ansible.builtin.include_role:
name: caddy_site
vars:
caddy_site_name: "{{ hostvars[jla_host].jar_lid_archive_instance }}"
caddy_site_domain: "{{ hostvars[jla_host].jar_lid_archive_domain }}"
# Tailscale MagicDNS: the box has no public address.
caddy_site_upstream: "{{ hostvars[jla_host].ansible_host }}:{{ hostvars[jla_host].jar_lid_archive_port }}"
caddy_site_resolvers: "100.100.100.100"
loop: "{{ groups['jar_lid_archive'] }}"
# Not `item`: the role's own tasks loop, and would shadow it.
loop_control:
loop_var: jla_host
# The per-site validate cannot catch a conflict between sites.
- name: Validate the whole Caddyfile
ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
changed_when: false

34
ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml Normal file
View file

@ -0,0 +1,34 @@
---
# Jar Lid Archive backup: the database and the photo store, as ONE artefact,
# dumped and age-encrypted on the box itself. small-backups-box pulls it.
#
# Only `web` is stopped. It is the only writer to both the database and the
# photo store, so with it down both are quiescent and are captured at the same
# instant - no row points at a photo the tar lacks. db stays up for pg_dump.
#
# storage is deliberately NOT stopped, though an earlier version did: on this
# VM's contended HDD-backed disk SeaweedFS can take minutes to come back, and the
# first test run left the app down for 26 minutes waiting on it.
- name: Configure the Jar Lid Archive backup on its own host
hosts: jar_lid_archive
become: yes
tasks:
- name: Ensure Jar Lid Archive dumps itself, encrypted, on a timer
ansible.builtin.include_role:
name: backup_source
vars:
backup_source_name: "{{ jar_lid_archive_instance }}"
backup_source_description: "Jar Lid Archive ({{ jar_lid_archive_instance }})"
backup_source_dump_command: /usr/local/bin/jar-lid-archive-dump.sh
# Uncompressed: the bulk is JPEGs. See roles/jar_lid_archive/templates/dump.sh.j2.
backup_source_artifact_suffix: "tar.age"
backup_source_stop_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml stop web"
# `up -d --wait`, not `start`: it also repairs a db or storage that died
# meanwhile, and only returns once /health/ passes.
backup_source_start_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait --wait-timeout 300"
backup_source_retention_days: "{{ jar_lid_archive_backup_retention_days }}"
backup_source_on_calendar: "*-*-* 02:20:00"
# Reported to Gatus as backup-dump_<instance>.
backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_{{ jar_lid_archive_instance }}/external"
backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"

2
ansible/site.yml
View file

@ -57,6 +57,7 @@
- import_playbook: services/xmrblocks/deploy_xmrblocks_playbook.yml
- import_playbook: services/memos/deploy_memos_playbook.yml
- import_playbook: services/navidrome/deploy_navidrome_playbook.yml
- import_playbook: services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml
- import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml
- import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml
- import_playbook: services/headscale/deploy_headscale_playbook.yml
@ -72,6 +73,7 @@
- import_playbook: services/forgejo/setup_backup_forgejo.yml
- import_playbook: services/lnbits/setup_backup_lnbits.yml
- import_playbook: services/memos/setup_backup_memos.yml
- import_playbook: services/jar-lid-archive/setup_backup_jar_lid_archive.yml
- import_playbook: playbooks/backups.yml
# Deliberately not here. Every playbook in the repo is either imported above or

8
tofu/nodito/vms.auto.tfvars
View file

@ -90,6 +90,14 @@ vms = {
disk_size_gb = 20
ipconfig0 = "ip=dhcp"
},
"jar-lid-archive-staging-box" = {
vmid = 3300
name = "jar-lid-archive-staging-box"
cores = 1
memory_mb = 1024
disk_size_gb = 10
ipconfig0 = "ip=dhcp"
},
# disk_size_gb here is the music library's ceiling, not just the OS: the
# library lives on the root disk at /srv/music. ~45 GB is usable after
# Debian, so roughly 300-450 albums in FLAC. Growing it means a Proxmox