jar-lid-archive: a staging deployment on its own box
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
97c7d580bc
commit
c3a7ca0ba7
GPG key ID:
883EDBAA726BD96C
22 changed files with 929 additions and 215 deletions
|
|
@ -62,6 +62,11 @@ subdomains:
|
||||||
# has nowhere to carry a second credential. Same call made for monero above.
|
# has nowhere to carry a second credential. Same call made for monero above.
|
||||||
music: music
|
music: music
|
||||||
|
|
||||||
|
# Jar Lid Archive staging (jar-lid-archive-staging-box, proxied via vipy).
|
||||||
|
# Public with no basic auth: the one-off photo links are for people without
|
||||||
|
# an account, and the app has its own login for everything else.
|
||||||
|
jar_lid_archive_staging: jars-staging
|
||||||
|
|
||||||
# Read by plays across several groups, so global rather than group_vars/<group>.
|
# Read by plays across several groups, so global rather than group_vars/<group>.
|
||||||
headscale_namespace: counter-net
|
headscale_namespace: counter-net
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,214 +1,218 @@
|
||||||
$ANSIBLE_VAULT;1.1;AES256
|
$ANSIBLE_VAULT;1.1;AES256
|
||||||
66356262646437623239383039643762363139653565356263623438303833663431396566353163
|
33653731643464333331646636613630383836323134363761333038383334353230663631376666
|
||||||
3631323563343638346464386661346235656130383238370a353236623366313561646334363639
|
3861376232363238613234663263646137336662666263360a633661313435616338383337396664
|
||||||
35346232376138666430363437346638326264646562386436323962633630663136626462343762
|
31316234616365326363353739323230326264623562313061303339373265663435313938326634
|
||||||
3862333139646636650a656337306533313436333239653561333231363030633339303835396637
|
3464636635386664620a663330396463653963613537653230333430326137333964386530343636
|
||||||
61633763313464353933306333383061346563623637383166646332303237653065346233623134
|
62383336343031363537383362643934643839653731346364636534373430303636353431646534
|
||||||
39386130613866386235376266643062653833643132633235393931636366376164306331303762
|
66313264343338623333633964376137623833613664393137343432666264656466353830346631
|
||||||
33393562303738393364323465653163633730633636343764646337383733643362396132303063
|
62616639343265646661373062376138363232376562316134303730323431336533306565646438
|
||||||
32336461396561663063633137613234356535383863346334666465643437613962376439356433
|
62666132613930646564393162316461643431666664356436356264633839333338356561333038
|
||||||
66353532333933343462636163353637613637356530333331636363616265376263636538656566
|
65333466396165363030653137663939303838663030666235356639616362346330346437306133
|
||||||
62643137623933363237343865613131383766633561636664616434393538646332346132336630
|
61653130363136643038626562653437303939393133613733326530656163323531386436656363
|
||||||
62333361626339626638333666393735373834396431343965386532313834353837336664306530
|
37303930613063366537623435396564643431346266373363333332663632663939346131373565
|
||||||
36303138373734343735653637663466353435306166316661353738353035366535333265626330
|
32663430643034663530643431346335623636663265316431336361363233383761333130353530
|
||||||
36396538353566316634376662383564323861303063386163663061323435656261316439336539
|
30373566316336646536313938353663666436643532303433643139376232646566616430366330
|
||||||
65323063343961646435373134303039373139333265633838313636373463316231633832306261
|
36646433393037373733633562306631623164356237326532653833633031633735653463343461
|
||||||
36623934343731343938306332356261646264613063393834336332666230626534653464363335
|
65646664383531346531353133313334663130393237353436623638333731326139653363383032
|
||||||
66356331666434623735613333303466656263313964363030316231303330303735373135653763
|
63356235646366333534366666383961636361626636616237323237346234383035623365376137
|
||||||
30323635303435656639653037366430316330333033306339366361353765656337316561366263
|
34646162663761343662353631393131313562326138336565383764666333383731353730376135
|
||||||
38613265363565373832636239396362616435303536646465326132626239353361613137626632
|
64656231613630646330643133373766643330656166346134363263373966363665323964396233
|
||||||
30353764323430656631366134373263343861376666346166363131333137376466393438346461
|
64333038373565343330626633323933333239623634333436666338633539316263616438626461
|
||||||
31613435363637306432343437653636356665373838626361306332316664343266356666353238
|
30643566656534666533373365613861383231373062613831653537623532353561633736633866
|
||||||
35303464326633326438353730393462373437313864326638323262323034346464643038386462
|
61383532633430323330646137613633383830393136303332643531383466353039353564393232
|
||||||
34613435326432663566386331333430663336656239353131633261626637613035636662346136
|
39326130333336663038643766303464363664333261313137663863616466666534383831613732
|
||||||
33626131626435316566393831333534393663326465633531373932626631613063666165613766
|
35303831633232306131636137656461653361346134373935333462306232323165313265323239
|
||||||
63636463656435386333323037656533306331646561363061613537663738326162656332393161
|
36343864306130303537386164653866656632633861363736336362656161356463386265336430
|
||||||
62313338376630643031666630663337646439623862306663303335303232306161353336643334
|
38323739336364303061363437346137633436653566353464323462643838313631393235643635
|
||||||
30386537336265383334336431653637303836376261303038653739396263666161653365356133
|
65626461643263656537616638623066346636663335383861376336623932343439353231343739
|
||||||
62636638393465363239616264316331316139363063313431323165313237646437643361653539
|
64663263613862626266616362313136373630636262633036386462666632623336343431333231
|
||||||
63393932623934313062376137626266363330623766646231353561646332353565643638356633
|
31333164356538616562353335313161353339653262616439636134383063346566663261323938
|
||||||
39656565363962653038393430363666643361343131323231663566343035643631646433306464
|
36633631636265396230313033373230373635643165613239623532613436376464646635326535
|
||||||
64343735393937333036623139303531343338373162396635613863316466393235343865333065
|
39333433336361343530623434356636363433653035333934653162363135626564346161346130
|
||||||
63353662313737353737353736316261346466366666666536653064356130326334326137383933
|
65313130393639393539633534653330343935376365656162383132643635313034383538333333
|
||||||
64306331663430313838376266613763343537613965303937393364396238663266363539313531
|
64346336346231313939396463643166363165393439626431386334393563646337383361326431
|
||||||
33616133646365396336613234636630396239316633333832653138393964616261643531663836
|
34346462333732643365303230383937316462623566346238393431656433323535376361623561
|
||||||
36666265386134383430626637643766636439393636336436366265396239303335306563653962
|
66333530346638623332633832623532393035343930663434343737643032303764343138326638
|
||||||
62323763626265373037343766386433646637633838643933633738333037633336646663316335
|
33386666656162393633636437656564326336343661643137323362653437323939663033316230
|
||||||
36373534663439306137353236343565636538636333313239373936653263366637613335663233
|
61643437633665643439663566663232643162656139383236356538663139333162383665343766
|
||||||
32306536643866656461343666396565373661353561393665643162613539633634666138626266
|
31326539313163333034666233316634396434316535313264313430313432313938383534663333
|
||||||
35613663376465626363373864626266393335396461316338653262316131643633666361333537
|
33653165303864353638303833646134363362306632336133306430353165363063353735336464
|
||||||
64366533633535313633306535393732383436333762616539646463366665623866323463336263
|
31303536333964333833343566383238636233366233656531343664346331386230316365626662
|
||||||
38313365326338386664363538633232343636303135356230393065613238386366323435633931
|
30316635353262303962316364386263636531396636306534306464336536343138663933326236
|
||||||
37353537313330396463383233616532653139376263346264623366383339306139386231636466
|
38653563633663356339633335376361326132356334626163616361383134306239643365343562
|
||||||
35383761373733386133386436316431613265306264656630353561333736613734653164333031
|
34323031383638313336663937663564376362663534373464386635653164636430643266343766
|
||||||
32643237366534643062653065333632343563633034663266393961353763663534353466383561
|
62326134623638343735613562366232653137613265653836326461386539646133303638613639
|
||||||
38373332323135353932343231353433366637333437636238623235663836393735346230623438
|
31373465353533383237393933626333383839313736636132646564666639336530386537663239
|
||||||
37616639643231396564373033636162616234366262333461343139363539316332333431636565
|
34653533643039633337353535393832343439346536336232373837363734366430336539373732
|
||||||
64396639383462386239613035303565376163633637653137373631383864353532373465626630
|
66646339616366313834333562303466373962656435373530666466383339653064666532663236
|
||||||
63636531653063306131623735393963623836313566643565386336636565633130383264316533
|
64393032623234623561313766343233643232303630613165383761323435313130303361346237
|
||||||
33343030316365373164663661306161363062646361646332393935373533626331623163323038
|
62323430353333383539613536646439336466633435383134626530326531643135326264376662
|
||||||
61316666366263313531373530366639333364336563623162323938356662626565383363303533
|
37393832643662386137613230633964396338616638313764373464616665316531396462326431
|
||||||
61323730313161633035646435346234366661386339346663363438663862636361343033306134
|
36666338666536613666643032376331626132316433653036656330353834323865663865366535
|
||||||
66396365653634373235623837666537323230616334663263633135316433313734663863333561
|
36313937346136353534373631396630633432313065666639623538343332636665313236336366
|
||||||
34326134613335346661316632353764343235326630316535306666326662613436383764646366
|
35613330336136623331346636646134316265613863323730663837303032303131376435643131
|
||||||
33616135333038333834656439383733656436663433303536313961333662366564376338326661
|
39353965376565356136613533623235303031646331663166396631336434623561643464656263
|
||||||
35393839643663333130383965663538393931363662336165636435643132353461343266363265
|
64376636376337336661363133663139343935613033333266663439326239646434633131316637
|
||||||
66346635313337346661646461386336386461346133323435643939616234666665616261313731
|
38643164383136343939663332653433376362363432333265623166643831303639376637366363
|
||||||
38383265643830646364373164656133396132666236313265346337643939613831303366393530
|
37336361336461363437646234663330393830393861373162663463366533366637646530373838
|
||||||
63333330346131656134366237323661316534323330306330616564393130353136306461666361
|
32633764373164646339653931653066333833363534663234663864336465333164353231663032
|
||||||
62376637646230623264363437313161396130616231346133346665303834623833663062663436
|
36333666653633643362343334303035656539636663306333613964663837366338613565626635
|
||||||
30643736376638303065356161366561353565343139343731313263393933313239363363636638
|
34393964303866313835373936376130626665303031376637396639353734373337336534376563
|
||||||
32663932626332626366333635616335316235363561373539623265663733373030373439373831
|
30643261663235323831366537613434376566323837633136306663663331333863643037336133
|
||||||
65333763653635613964396437333833626535643666643433393031336461366562333934666234
|
63623035376233393431373361653964323833396362646661343235343332636435356365346330
|
||||||
62383632303436633965643430363763353361613762306361613066356136383131383232313961
|
39663533353631643062613839643761633866373539613536326132386336373731633239323362
|
||||||
30626531353933336563656139316338616565303261363933306164393966343037353431653233
|
31383863333330366665393666363864613632636537363436613230623030353436373061633037
|
||||||
32636631373966323838333666313964323933306638653162646166623237343662646134383337
|
36326633356533666438353034343966366136316331376431396536393261383633373131656530
|
||||||
37383039383531656337626334386638613133343133633637343738343731326663356666333161
|
36336364653962303730393338313435316162393561383161646232663430616239623964623431
|
||||||
61323765636135306563323064346465633537613361383761393361353363396237333438306166
|
38383563386266343935666463636533353361623965326232663937326462366433366534306534
|
||||||
35386338356135633365396661356535623232613765653931303063353538643130323439366438
|
66323139346238343031313762353166646235313463363435393236633337303561333930376439
|
||||||
39363936316263333562646133646464346362343761336662353764613261396364626164313664
|
37613630386636636634396437623866326639343863353134346438663033353232646566336164
|
||||||
31383839303361323461663662393736616363646232666633323134653138306161323735386638
|
63376535363235653362623630636661636162313462303664393032326538613139653039653065
|
||||||
30333365663064366535656134323338663566616163333634353939363533356364393563363333
|
38626234353034643330333861316665306665313933363234623639373765303065376632643233
|
||||||
65623838323034336530326363383237373563326334343066323738663833353930663930373965
|
39633764346132356564333236306333373664366134363037633730653732643130616665343466
|
||||||
30363766633137396561376636636566363862353062663038653534623430643766633631356562
|
62356138656331626437666465663132663562336365353834353632383062326164356161366138
|
||||||
63623762343433626363353239376539633136333038623338303961373938396632313966666532
|
66663532373464316264366661343863363265616438323132646361346135653063303031323833
|
||||||
33633865626462636137343436386661353561623035656435623566613530623163316465383664
|
33306231353231316165373137616134326163313430613963653532316331323738633639356565
|
||||||
37346261616239653138623262623930316432633531313037353938396164646534316564613238
|
38343537616330376263636165396139396365643663393637626132306564613263323362663637
|
||||||
36643137303965316238346134323639386261633338363130323134336632636564636638653861
|
64356665646532326466393733303130386230363831666265396635363630336331326662363864
|
||||||
38346566643966386465366535393166626463313039613236393831393732613537336530393936
|
39383432346265376137636639613162643135343433303435336436376161303766383232656166
|
||||||
31613234333836333732323165653931336565316564346239336663363736323763623962383261
|
32323462313432643330303039373239323939626266383232393433366232636134643636646431
|
||||||
65626332626565636162336264303964323139303831656262373161303861393366353237393136
|
37376330343461393766643664393864633265393631383566396530393364346634626535373133
|
||||||
62303361636131646335633733373966626135396630623133623733373565306234646165656138
|
61623931386433396630616463353236393630343838383865666466383632363631373136393930
|
||||||
35383065613365363438396332306236646432663731326666613838643434393866623163316631
|
35633934393963626263646137623262666335306233356439633334313765353161653566613833
|
||||||
33633931373135336438643735646266383462386431666330363066386538386335383231653334
|
32346637393530313730376564656434393737383037613664666338666435373066303338623930
|
||||||
64323366363032386662333662613562333864356238626161366566356564343339343838373437
|
63333763633533373437363666616263646564613466613830336134353933626336343036646137
|
||||||
30363133363136663036656461343534666636343932313336336637376665393564313234373430
|
32633265636134333438386330663764633432376633383930333835363362613738353464636330
|
||||||
39356362616539366237646363646364326664653232313235653831663539353530393261346532
|
38613135326635643436346565633833623630623736623231306237313662666666653532333234
|
||||||
38653134623732313466386635313730303639613561323665356633653836393237333737323864
|
62383866373939326335653461653238346336636636383961613534326336653030386534653934
|
||||||
36323736326239636565346536646566656138666630396366353930383237333338626532663131
|
39323266373664346437613362373162643532636566623563303139353430343638656130303866
|
||||||
64366339303230663530626632623066613030663533366539313866646466373439663831363936
|
63346537323431613434643865353865633331653233343839376237646337613566663130306537
|
||||||
36366636653032343064383661383664636463343165326532653861663537633238323937363839
|
39343735626236653636393465633038323465656661626561656438373835343361363735386638
|
||||||
65333030653566306465643035633162393534323731346330333861353236393732346130313034
|
39623830303864636333373534396562623630653362366166386264643438373435386438383831
|
||||||
64303736323030643533633539623638643232306435663238393566376564313162633334613462
|
34633264613230643434313037323538346661346464393362613434303135313136376131666530
|
||||||
33363931383664633433363032643536356135626662653733303339656339373735643732626535
|
36373866313739323962333061343836363661346334383033613534393666636631363262323238
|
||||||
35316131356135393933393462336163373062653133356133393830393638313838396361336635
|
64313330316464336533663764386666313732616338393162393434613437623730613038356663
|
||||||
62323463333464353939346233333462646161633036643332313337353065376561336630383163
|
39383236316664313731626330323039373937366638373461373736633239666438306437666666
|
||||||
38306666666634393766366436653436623138363830343636323833353365333762343533633835
|
66316639346432316631616261626262666238346530353563356436353332313362353431646430
|
||||||
66363730383932303661313831663566613830393530616164323963636339393339623065316534
|
64643366656466333363313135653331303764633839316239333630353665666465393263633333
|
||||||
38373662636330336632653335316463623935653961363566396132616433633831636334306236
|
31306663326366643032653762646464396530643662303636393432373966393934396634343032
|
||||||
34393761663962623563333534383739613965353033646235653532303330343961656639333337
|
31346365633933313436643937626634656535323431616231393866663334303632613237633666
|
||||||
31643462663538313930383038343964326463363935396365343466643236653738626639616434
|
65396262363665353837366166633435326133356330383066613537653736656565306434303663
|
||||||
62343337356239316162383564666131663535313633373062353030336439396533376235386131
|
38313837303965633236366533386661366464613262326162316365626534356237663035396264
|
||||||
31653865643931323364356566316435333336316166643063613164643266313834323235353863
|
33356165623232323664363731383962616465653230373234323961646236356132383031346465
|
||||||
34623936613366303161353165386662376465656366656561613466383830326162336238656236
|
61396136636137303933346462356339393236313361316465346631666432626434636366396531
|
||||||
30646632396231653263373631346330376461303231373733646231373462623131303633616166
|
35313264323766663436666430313864303065313633396631656438666564313038386161313433
|
||||||
37366435353830303234393962663334656335356661333437363637663664323430623135346262
|
66383734373435633838666463643636376366633334626130333730366639663961666630383234
|
||||||
33303233343362336261633663336635366233313466636233333836663464633637383964313133
|
35343964363931663431373162653333613563366462333037363838623662303837626237396132
|
||||||
31363461336237373337353438396132616134303432336533653737333538383534363735316334
|
36366162313766366237306337333338623937363165306433616530326435363063333938393939
|
||||||
62303666363532383131383833383536333733306163366339653038313930646531653565386335
|
61373433666233393962633263386534333735303564326435623363663334313163326239653966
|
||||||
38313561303062336232333634343866393763636131333665396430306432373734613834316636
|
30656132353938326530306663613637373838323432323762306263623136653937383465613038
|
||||||
61306138626530616230343462613036323166656334656637616139663030623932386132393762
|
66346661646231623161373233653337353162636334376137373138613864353036663964336362
|
||||||
36363364343737613165376365323361363233616532643866303632653063343431613338373633
|
64303362306239633339313666303636626264353833333333613132316261333830616232333662
|
||||||
63303263626434326539316439323239616632653632346538336535373336623739323931623462
|
64623061633466336336623734616632303265363630356665303663636334656136636339353534
|
||||||
35656365336630323035616161656237353135373366376466326264323863336166633438663834
|
65636564636462346133666438623461653630333833643230623632333266663066376461316661
|
||||||
63646562386339363235396163646334333238323932383230643964366466323064313138323464
|
37623637356333386638343937613361343265663033326337666138393233646639656232333864
|
||||||
62636239663635336332633334656631383261306437333963363464303430666462633333666638
|
37343733646432353564316237363964656532386561666538633737323739383766316337386137
|
||||||
37633830663439396133323861303735346561376434386131363864316332623137376139376231
|
65656231336463623965653232323162363866373732656534376430356432653539393939343661
|
||||||
30303765646132306261363335366565306564373261663336613336336632616231363366316135
|
30626130336436343033386637393334363237373737613761363633316533623033343731356338
|
||||||
65303966616439643863326662656439623938373833323738646535636230663136653266653533
|
38363231303637323861376662356564323032316130356461653466336165363363383637313862
|
||||||
63326235376633333366373436333534336436366434363461393763633437616262326133623563
|
35666338383164666135633734386432353137363464376538663465353330376237363333653566
|
||||||
37613161303161356632316335616639373564306161623339353066383163303431313333306635
|
32376462383732656135333861323465396664353032393432373861353836376239346338393230
|
||||||
36326435376661343237303633316462613532313232643461313065376134393338613130336161
|
39623532336232343761303235393963333065653935616230656236646330323465303263663236
|
||||||
38326463303562666662613361656161306139653431386630303561626666613037393238313463
|
39336233303030656161343961353566626462616333643235623032623338646163346564383034
|
||||||
62623533396532326332326636346134616236653638363837643131343166333564323538653638
|
63333439653936653731653337396231396562383565363837643138373864613338386164346262
|
||||||
63383261383265353331663864643933373562396565346163396434383765343962336363353261
|
31613133303833323238343665353238323335623162353362646337336335323639363730386462
|
||||||
38353034643436626632623235396261326235323937646135326532313561396361363062333366
|
61633966313435313165643537626330663864623762633330383730613131396463313231343837
|
||||||
32343666343163636233646466613633383034653438646262666133643830333834383233343136
|
61623965393836303230626463363438343239363562643334653733343865313334383233343932
|
||||||
33646437396137613466643031323538626332613066333065643237353263636435393964356464
|
33313930666136613766376435636432303466626565326434636561373565306532376239643735
|
||||||
34383265303364663233326432313230336436373866303435336666343832383737303030306534
|
63613037616534656265663536373164643866653466326361313965383635626331636463383363
|
||||||
36383036363937393965386431343561333961663565333565333261323033616630343130613934
|
35356533393461376132626431383839393132663161373831366130663964313839613566643165
|
||||||
32383166333063623235333464343864326531333165346261643131393264346534393035393431
|
63353139316333666337636635303236643034623830353462626535623064653065316135343564
|
||||||
64636139383232383332623866323363623666326361386134353030376366323039393033653061
|
64363963383230313765643262396330663938366631623834376532643937316236623865613665
|
||||||
63383338303734323637386366646365663032636463336261643533663638343661613462373736
|
34653331333336333364396330646632326266626638316566363539313532323762383034323036
|
||||||
32663566666133656165396233643334343335333731333938376236633330366262366139326133
|
38343838616264333737376432366535643339323734316563386439313339653763343139306565
|
||||||
32653163343963333739303061643936363861313938646164316564393261663765393462613837
|
61383538613761353733653933316634316138313565393932623638636135303962366562623231
|
||||||
31646636326330663464363034376230393739343961653130303166366233396262656634313061
|
38346365616434663362306138373634613334613037613961383032326661356163626238666233
|
||||||
65646664373161356134616165383265346633376136376132643234623334353563323539333065
|
31653938633938366632326439333237663539383461646236306265326536626135343161396137
|
||||||
61623265343730326164373063383937323766363466343466316364373935626138343938323261
|
34616636663639396534363431666464306538623632623133623133643864313839653865623266
|
||||||
38323739663131346138623663393832346236356631633133313436363364663536316561623631
|
35643865366465346532386130353565633333613061396139383265663734323139663532376638
|
||||||
32393562646666343537663039363165646337653564363065353433643031353865386361613534
|
38623966306639363938643564343330636363346261333166613938623936303966613037663365
|
||||||
37633639303636313261613563353136346465326163623361323230306364396362373966336262
|
62376334613363323563363332356463376532323964333338623239326663343031373635663032
|
||||||
31353565623239343564333533656236656666316638646666376138373164663138633835366162
|
33626432356662636130363761323862616131653630396164623435333061366564626236643936
|
||||||
37633036346562663261343731366161613361656534343839303062373232636339643961666432
|
65316562633036366233303538373530343932616533346637656464346166306438396435666533
|
||||||
61396665656638333435386338383638323664313139656331336131326563373235646539633835
|
37313766366636336535363536333061373264663163303162666330633061356464353164636239
|
||||||
30393235316564656131303932386336366661373737393264613364663030363862656462646463
|
39623836663066623431343835623264373239363766323338353633336338363536643462666236
|
||||||
33303766303166336132626265653431313263363131653835336634356632666230666235383262
|
35313031633561353066306336336362643863393730613938373539363831336266353266633036
|
||||||
31653837356536666266383562633437376161653664353533656331303563626539626161653562
|
30313039336631396664383434316462643366643532353638396339353430303063303734343830
|
||||||
64333337626430616330346233613566353330383361353234376236656336653735663838343333
|
31323761643235663061336237333866323139613533366463353433613832316232323938323331
|
||||||
37643162376630373531396138643861326562323133303363363837306138373434306364356364
|
39356261636231353936313033363962353232663831316634653634323965313035663537323864
|
||||||
64623037643737663138626262326630323638376336663062393930663566316632323862653135
|
33633138383431376131616462363337313035326339646661643730666337393266316562333138
|
||||||
66363664336431373966663133353030393863313538366632643932363438613035326563323163
|
39616636323630356631653834313466393533333761666534333239306666356166363433326434
|
||||||
63343565373862316230643166336438393464323032666334653631393761303337303861666134
|
33643566323438333934363236313633666465663236353034653439323338613433333538336535
|
||||||
36663637346136623564613233343432383930616339626161623933323565623861653939326165
|
30326435383362343065356463373761333565396131336562636236353863613635636436326131
|
||||||
37383436353533613736323237666433316130613464396333343965393031333138396461376632
|
37353238656337633135643430306532393530653032663437303139323463613136656563613633
|
||||||
61373166643766376234646436653566313637326434363261333831336136306137333061323035
|
39303463656631336231643733323439376331653732376639303231333533373264623034316435
|
||||||
66303335363035666366386364303861383231366339643666313636623733616562333539643465
|
64386635636565333961663936356131643536636235666433356338646531306535653366363362
|
||||||
32353566326262326637393434656133313963646435333135613630313865373430366265666632
|
64336634396666303364663033383034616639646532333238336138343032376631326139646664
|
||||||
37316539363439663131323835313761313765653564663337346462653332613430373037333064
|
37323432613664343032313838663561373735323938666339336133653131383263366665656537
|
||||||
64323731623566326664613634626265383661346462323463303638346464643366663333336339
|
61363139356666303438316434626536323362306539613930353263663263353933353664393666
|
||||||
65626265663230346338633764633436653339373031623438633638326137613635633239343834
|
37663763616635306665386163333034643465376130313861633435366666653535383234363438
|
||||||
35373132376133363861373062386235353836633030323236343666323134326135336231613566
|
34326565373461343035616162316532333739386562383466626436366434306261363931653464
|
||||||
34326263643163633935643065306165643564663831633561353061653331326566633664376432
|
63306238663038373631363834313261323337393263323362306636663765393535303433643562
|
||||||
33353835366265346266663836326465356166626432356663366566313962663862363930336334
|
63616336383563343732653034316133343662333733323231623064386561623535376235336432
|
||||||
65323337386231376361343933353865633330613466323337306137633762653931313331313339
|
38646537303136653939346335653031636266343132626262643439363564613132356634666435
|
||||||
36663965353832386662666433383934303737656336383432633632646135383835336631373861
|
34626562336630363363393261643735353263653462353336383630366335643966396434363034
|
||||||
63646164373137316662373330333465343134633466396566333334663036663039373934333832
|
65623233346335343637633564393466616365626361613134306362663333353730336264616664
|
||||||
35343661303565646464373834646234623837363562386463383237373533316638666663613633
|
33656361323631663639303033663632663262313461663133646663633833383833636661336463
|
||||||
30313437626231343062636437363836386635333862373362643532646335353863323936663034
|
37633266616639663830343333336362653838316363303637333535663136336633393339666263
|
||||||
30316461373330623130353563643035303561303666393464636533626562633165393236663936
|
37313434303638616530336366303865333038326534656135613136613930376635363062353962
|
||||||
31313061303465633332646363643866306439383636356366623565663437313365376331656666
|
38323339356235663965383633323665313932643632363132386666363239633666313566653063
|
||||||
34333930333235306636646664326561613165633233353937666238646536323131313232626366
|
64643433363234396563356435383335623133613364313834396236376339633738326235363737
|
||||||
36623237613635343134313366346661656433643136353433393532303165326564646331333035
|
31303566613461386434646233306261393835613739353831306164343164616432656466636139
|
||||||
35353365633039353161663337306463303535643062366232336239656162653532306539333035
|
62393132636465663965633432353665643535643233313738323236383166653838663338323462
|
||||||
65373735373832326337333962643330363338656134303637313337393466366263363264343631
|
61383035636632323863333361643333363238636436616562393465326230373164366261313564
|
||||||
30656433643562393537326361323733643639396537303065623936336230346664616534323262
|
39343861333562373762623437613334396634613462373132386564616462333638373161653737
|
||||||
36313864396364353836373538333130303731383437376137383938383633373063323936613464
|
38313366373864333064646133393561396537353333626462666361326532323239376431326330
|
||||||
37613166323432363232313834326565663265336138396335313230303935396266646438323833
|
30373734346566663235616235316633636263373431326166623734303734383337643562666536
|
||||||
65306666636338373239346332613464373235613137643133626261313031326462356138356632
|
61353130646537623936646639393439306639356130373431623431643834386632616165383133
|
||||||
66363465346165306533613636383938636531656631306139666237396632646634333937623032
|
61666165396634613763306561356533663263386639643766306166303162376366363530396361
|
||||||
33613439656637383537356364336265653931613835333734383836343930313838626564333065
|
65356538663638633261636332386262343236663731313932656339323239343938666662343064
|
||||||
35363735313162616238643438316535356663396339316331633232393232316131396432653639
|
66353234346530366163656437356464376136663261343938636566333865313261396164316137
|
||||||
36303335643534376536383331323562353139613130383330636334636261326462656565643138
|
38663238656365383131363239333537646333656136656465636464656530306137363436636231
|
||||||
63326365363163383932653562363833643761653839306131343737353633313737353732633463
|
39366339663630653461383461323234363936346230643266393833356432396661356263333238
|
||||||
62326563336339383661376665353230613366646132653364396338663337326136333263363838
|
38656530306133363733336335386238626132356237313334633634393634333937643631623963
|
||||||
35626537666238383435613431666365303133373166386136353061373765313632343665306439
|
36326364306664613832363734336463303635373262653632356263336665353739373231316664
|
||||||
34356139363636623639626633623030363933303539626665383733363638303832333561653130
|
34633036383439393339363364346165376261613339356236323330306432616665343265363564
|
||||||
64393437386362333731643131663263626434666661326533386362306362653732376366626139
|
66666137346231383465353666633835393737313261643935656530356162613538363864343830
|
||||||
65626662323232636134653334366639386164653535323435613863353332366630303636663339
|
35323865613563366130323132326238346338303761353433656532643131333962373066336233
|
||||||
34333633373737333732363664353437613762383064316134393139393539306564396237343332
|
34303734396235303137616362653037333138306132393537646531616635636561343831366531
|
||||||
64636339643030353365356136366264353935643830633237316532316265316635393664326533
|
36333932323234643237346266323466303736663665363362373465333139393934623939313535
|
||||||
64343030396334326135336465336637623832363065353163316465303537643861306632336535
|
34316433323136316533313663393238663530386565376131366631393031643362346239316430
|
||||||
31366338323263313965643565656430366566393130326333383862313239356437623232613865
|
64303636343765326465616464663234373137393137666635306261653466323733303962393461
|
||||||
63383863316639346632323834633739356464313865376463323633393365653033376431373138
|
61356438316631333231636465653933343264356230313937636431336261306464316532343335
|
||||||
61363035653233373036666531303331383337353839363563386333313730363664666463333735
|
32383732363866383531373136353331623566636332643332636237653737313064393566373837
|
||||||
61383638363164336265383664376463616536383533666265383961353961303833353265366633
|
37333031643034343637333832363931343938303764313836373230353931613535313765393065
|
||||||
32646330333336613631633830633837346332653437356536363532396438326364623465383133
|
30623234653039316461626635366337633265623236333963323934313065623732353362303338
|
||||||
36656231343734643830383335623930343132386336666366623161666637623862363033373236
|
64663130383030613536383739646562636662656234363036613162666461373834333530363363
|
||||||
32666332316266623230666231663162643262653231313965323137383532633663323239396139
|
62383166353333663232363737653639316235303535616536383962653039386464303435356538
|
||||||
37656533373065623161623766333335353965373731346639613566343236363239313537636239
|
61653166633162616563336663323830383732666264393262323139303936626464373132303738
|
||||||
34626533346138613764346161383337623265353563343066303236333534316137356636396565
|
30343061313535363838653930306164303438353736393862643461363333396539326633386236
|
||||||
32616136306665346566616565346162613736333836623363346232373932366365663531653435
|
63623666343330363166373037636332313966303565303061663361386239353962666164383435
|
||||||
37623539366233366539396361386163643161653731353232313034613135373039653339346637
|
64626663383735393431623436323862656262396461393663616530353132646234353838633537
|
||||||
31623138326133366333656661393837653939343064656638616630346462363065623139323131
|
66313437306330653963613663373235383338383439636563643333323265626132326639386639
|
||||||
65363036383433663566626336353432613535363239373264356237323032323634383662613763
|
38663762326666343935393664643761393236323266336265393033643734333439353538613062
|
||||||
39646161363639326330346332353965366330346632303939623136303238323431346361656462
|
38626438663431366464343937316165313937323732643061643334643437353764333231646362
|
||||||
61623838626139636163353430636461666239343337373132643334616234616339313761353331
|
65393566663530383133393430383237633638363363313963383230663434353635353364333830
|
||||||
36643165373834373437363165303861306135383236303030626463373235393234313765343563
|
30306535353962633865346131333035353638336539336635643238383762303563323663616164
|
||||||
37353932366430666539623230373161303935633565323366363232336264643961383835333066
|
63356630393664653637613935363536656239616239316337353730666632633234613331336136
|
||||||
32623366663965316662383036306462616465353139336333643936363838313538363832363330
|
34383036616432306533663164316163303332313239646662656536333564643539343533383939
|
||||||
33313861313636373831656431383434363664663734626363646365666135376331366335653064
|
32303935313565626661393334303564633136323164623965663364323966346636396532336535
|
||||||
30386235313037363365363232316338613634643235383037393635333166346632356435373039
|
38373232616265346131633930333134666333663961383063666237653462623635323537393837
|
||||||
66663862373939346434333130383065386135306661353066343330666563386136376562316134
|
30373438616561353430633139323163353138363735636530643630346566633631383931363638
|
||||||
63346435643132316434363666343964303335663131383831373430653632663638396333393932
|
66633266643762626530613238343663326236363134316337653861376561383631663937373833
|
||||||
36383664376666393761643730306336333538376637633135616166346432636464336434323031
|
32663063356164333731343263393837386332623665313135396665373436323366613331363333
|
||||||
3966
|
65393533353933643134643362366330653261313064663963336539653134396437383430313666
|
||||||
|
33663234616461626630646563616561313139396630646461386338623266666434376335306435
|
||||||
|
36643430663131373064313464633733356131386338643439653032353436656661356366626533
|
||||||
|
35343434396361303634636438633036386563383664646561663566613339323865363136333230
|
||||||
|
37616265613030633061376639386461383062353135666534346632353064353231
|
||||||
|
|
|
||||||
28
ansible/host_vars/jar_lid_staging_local/main.yml
Normal file
28
ansible/host_vars/jar_lid_staging_local/main.yml
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
---
|
||||||
|
# Jar Lid Archive STAGING. Secrets are in vault.yml beside this file.
|
||||||
|
#
|
||||||
|
# These live here rather than in the role defaults because three plays read
|
||||||
|
# them: the deploy on this box, the Caddy vhost on the edge host (port, domain)
|
||||||
|
# and the backup (retention). A role default cannot reach a second play.
|
||||||
|
|
||||||
|
# Names this deployment everywhere it must be unique across hosts: the Caddy site
|
||||||
|
# file, the Gatus probe (probe_<instance>), the backup artefacts and the
|
||||||
|
# backup-dump/backup-store endpoints.
|
||||||
|
jar_lid_archive_instance: jar-lid-archive-staging
|
||||||
|
|
||||||
|
# Fixed by the image's CMD (gunicorn --bind 0.0.0.0:8000); changing this moves
|
||||||
|
# what Caddy and the probe call, not where gunicorn listens.
|
||||||
|
jar_lid_archive_port: 8000
|
||||||
|
jar_lid_archive_domain: "{{ subdomains.jar_lid_archive_staging }}.{{ root_domain }}"
|
||||||
|
|
||||||
|
# Staging data is disposable: three days locally and on small-backups-box.
|
||||||
|
jar_lid_archive_backup_retention_days: 3
|
||||||
|
|
||||||
|
jar_lid_archive_seed_users:
|
||||||
|
- {username: alice, email: alice@example.com, password: "{{ jar_lid_archive_seed_password_alice }}"}
|
||||||
|
- {username: bob, email: bob@example.com, password: "{{ jar_lid_archive_seed_password_bob }}"}
|
||||||
|
- {username: admin, email: admin@example.com, password: "{{ jar_lid_archive_seed_password_admin }}", is_superuser: true}
|
||||||
|
|
||||||
|
# No systemd units of ours run here - the stack is docker compose - so infra/401
|
||||||
|
# has nothing to watch. The probe in the deploy playbook calls /health/, which
|
||||||
|
# checks the database and the bucket as well as gunicorn.
|
||||||
34
ansible/host_vars/jar_lid_staging_local/vault.yml
Normal file
34
ansible/host_vars/jar_lid_staging_local/vault.yml
Normal file
|
|
@ -0,0 +1,34 @@
|
||||||
|
$ANSIBLE_VAULT;1.1;AES256
|
||||||
|
37383535306133363031316535656461613038653562373839363432623163356234303061643566
|
||||||
|
6533333535613064653336313664643063326562666639640a373530386539316363346332663137
|
||||||
|
31303230323065623733663666383039346337343239646465393166663239643836356639373765
|
||||||
|
3564313065363266360a643239333330616463616533623435656164386264303634356134303831
|
||||||
|
36356131323637363734316161383065653663623732386638363165633265356362386538643039
|
||||||
|
64636464323730316638316430643339356662353736366332663738656566663137366166393735
|
||||||
|
34633430363931323330393234336661383532653366633735313365363562396137356438353330
|
||||||
|
65313737313131326361326631363864396536343633623963303632623465363964373964393430
|
||||||
|
61323161663939346435386435616364653336646165356232346335353235356339656336306138
|
||||||
|
65613636353636623562613362633761333637613066663838396638613161663532393030303035
|
||||||
|
35316464356539666631636265646433353537396564366538643962373364383730626164316265
|
||||||
|
63393034663835346362636564316135326430633963393139646365333666663630336462323361
|
||||||
|
31353830633335366635366435643864313665366563396134313938666537663232343162383366
|
||||||
|
34636231636261663739383366363962366438323962393732393864623238343961363035333866
|
||||||
|
61386630396334626564306536336561333265383464623730386263373132613438396332356631
|
||||||
|
30386435333739653637633032313966626437303634613838653735373666343565306663376230
|
||||||
|
61303734623865353835626464613033346437623234626136353664303434386262346634356463
|
||||||
|
34396433353431393935386136333331643539633138636135353339323930386634393639666665
|
||||||
|
36643462653061363062383039313936303062393364353366623965313530303031306336396336
|
||||||
|
65643663343163636338393666386134366337633064376530303463393737396165663930383562
|
||||||
|
64316363663636383764656139366433393766613262613433623637393538633130616232373136
|
||||||
|
31376636633666626365376533356262393065656539613266326465643663383439373661393631
|
||||||
|
30636136363066653936653837376634356333353036633165316662383337636435323162316430
|
||||||
|
30316331353361396132653233623034376539323064363135306262376638353664643163653334
|
||||||
|
37643638323038663531376336313064383238366430663363353733356530393661646138303230
|
||||||
|
30633862626166653362346632626261636531373130353034656532386230616330643134383631
|
||||||
|
63396138326336626431656130306531626539323934386332306361396239376466623866613364
|
||||||
|
36666632316331326361373136326465323365653034363337313135656262623465306439646362
|
||||||
|
30636639663163643430363165316231366232623332393635623461306339313762323265623732
|
||||||
|
30633464343235323566356434376535313238663533633630356163336532303765633366316166
|
||||||
|
38633462303762323930393337306265393033653061616337646163333335333737666332666632
|
||||||
|
35613132646636333335313263343136323337616662623465653939336637653134383766336265
|
||||||
|
35633638663636353537333238643236623033636163333164663062353863643164
|
||||||
|
|
@ -37,6 +37,7 @@
|
||||||
- {sub: "{{ subdomains.monero }}", host: vipy, interval: '24h'}
|
- {sub: "{{ subdomains.monero }}", host: vipy, interval: '24h'}
|
||||||
- {sub: "{{ subdomains.monero_explorer }}", host: vipy, interval: '24h'}
|
- {sub: "{{ subdomains.monero_explorer }}", host: vipy, interval: '24h'}
|
||||||
- {sub: "{{ subdomains.music }}", host: vipy}
|
- {sub: "{{ subdomains.music }}", host: vipy}
|
||||||
|
- {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy}
|
||||||
|
|
||||||
# A public resolver on purpose: this must test what the internet sees, not
|
# A public resolver on purpose: this must test what the internet sees, not
|
||||||
# what a local cache or the tailnet's MagicDNS happens to answer.
|
# what a local cache or the tailnet's MagicDNS happens to answer.
|
||||||
|
|
@ -75,6 +76,10 @@
|
||||||
# API the mobile clients speak cannot carry a proxy credential, so its
|
# API the mobile clients speak cannot carry a proxy credential, so its
|
||||||
# own login is the gate. See roles/navidrome/README.md.
|
# own login is the gate. See roles/navidrome/README.md.
|
||||||
- {name: music, sub: "{{ subdomains.music }}", path: "/app/", status: 200}
|
- {name: music, sub: "{{ subdomains.music }}", path: "/app/", status: 200}
|
||||||
|
# /health/ through the whole public path: DNS, vipy's Caddy and cert, the
|
||||||
|
# tailnet hop, gunicorn, and the database and bucket behind it (503 if
|
||||||
|
# either is down). "/" would only prove the login redirect works.
|
||||||
|
- {name: jars-staging, sub: "{{ subdomains.jar_lid_archive_staging }}", path: "/health/", status: 200}
|
||||||
|
|
||||||
# Ports published from the edge host by socket_proxy.
|
# Ports published from the edge host by socket_proxy.
|
||||||
public_tcp:
|
public_tcp:
|
||||||
|
|
|
||||||
|
|
@ -55,6 +55,7 @@
|
||||||
- {name: monerod, host: monerod_box_local, heartbeat: '26h'}
|
- {name: monerod, host: monerod_box_local, heartbeat: '26h'}
|
||||||
- {name: xmrblocks, host: monerod_box_local, heartbeat: '26h'}
|
- {name: xmrblocks, host: monerod_box_local, heartbeat: '26h'}
|
||||||
- {name: navidrome, host: music_box_local}
|
- {name: navidrome, host: music_box_local}
|
||||||
|
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
|
||||||
|
|
||||||
tasks:
|
tasks:
|
||||||
- name: Build the probe endpoint list
|
- name: Build the probe endpoint list
|
||||||
|
|
|
||||||
|
|
@ -21,6 +21,10 @@ monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=cou
|
||||||
# key was still in known_hosts and SSH refused the connection, which is the
|
# key was still in known_hosts and SSH refused the connection, which is the
|
||||||
# designed behaviour described in group_vars/nodito_vms.yml.
|
# designed behaviour described in group_vars/nodito_vms.yml.
|
||||||
music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
||||||
|
# lan_ip read from the Proxmox guest agent for VM 3300 (qm guest cmd
|
||||||
|
# network-get-interfaces), and the host key checked against the one the agent
|
||||||
|
# reads off the disk before it went into known_hosts.
|
||||||
|
jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1.151 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
||||||
|
|
||||||
# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
|
# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
|
||||||
[lapy]
|
[lapy]
|
||||||
|
|
@ -63,6 +67,10 @@ memos_box_local
|
||||||
[music]
|
[music]
|
||||||
music_box_local
|
music_box_local
|
||||||
|
|
||||||
|
# Staging only for now; a production host joins this group with its own host_vars.
|
||||||
|
[jar_lid_archive]
|
||||||
|
jar_lid_staging_local
|
||||||
|
|
||||||
[ci_runner]
|
[ci_runner]
|
||||||
forgejo_runner_local
|
forgejo_runner_local
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -30,6 +30,10 @@
|
||||||
- name: forgejo
|
- name: forgejo
|
||||||
source: "backup-pull@prd-vipy:/opt/backups/forgejo/"
|
source: "backup-pull@prd-vipy:/opt/backups/forgejo/"
|
||||||
retention_days: 14
|
retention_days: 14
|
||||||
|
# Staging: disposable data, so the long tail is as short as the local one.
|
||||||
|
- name: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}"
|
||||||
|
source: "backup-pull@{{ hostvars['jar_lid_staging_local'].ansible_host }}:/opt/backups/{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}/"
|
||||||
|
retention_days: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_backup_retention_days }}"
|
||||||
|
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
# Register the backup checks with Gatus.
|
# Register the backup checks with Gatus.
|
||||||
|
|
@ -62,7 +66,8 @@
|
||||||
- {name: vaultwarden, host: vipy}
|
- {name: vaultwarden, host: vipy}
|
||||||
- {name: lnbits, host: vipy}
|
- {name: lnbits, host: vipy}
|
||||||
- {name: forgejo, host: vipy}
|
- {name: forgejo, host: vipy}
|
||||||
store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo]
|
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
|
||||||
|
store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo, jar-lid-archive-staging]
|
||||||
|
|
||||||
tasks:
|
tasks:
|
||||||
# 26h, not 7h: the DUMP is genuinely daily, so the window cannot be tighter
|
# 26h, not 7h: the DUMP is genuinely daily, so the window cannot be tighter
|
||||||
|
|
|
||||||
121
ansible/roles/jar_lid_archive/README.md
Normal file
121
ansible/roles/jar_lid_archive/README.md
Normal file
|
|
@ -0,0 +1,121 @@
|
||||||
|
# `jar_lid_archive`
|
||||||
|
|
||||||
|
Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive)
|
||||||
|
as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and
|
||||||
|
SeaweedFS as the S3 photo store (`storage`). The same three services as the app
|
||||||
|
repo's own `compose.yaml`, so staging rehearses what dev and CI run.
|
||||||
|
|
||||||
|
Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to
|
||||||
|
every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local`
|
||||||
|
(jar-lid-archive-staging-box, nodito VM 3300), public at
|
||||||
|
`jars-staging.contrapeso.xyz` through vipy's Caddy.
|
||||||
|
|
||||||
|
Per-host values are in `host_vars/<host>/main.yml`, and secrets are in `vault.yml`
|
||||||
|
beside it. A production host would be a second inventory entry with its own
|
||||||
|
`jar_lid_archive_instance`, domain and vault, not a second playbook.
|
||||||
|
|
||||||
|
## Updates: follows `:master`
|
||||||
|
|
||||||
|
CI pushes `:master` on every green build of master. Every run of the playbook
|
||||||
|
runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only
|
||||||
|
if the digest moved. There is no watchtower, so **a merge reaches staging when
|
||||||
|
the playbook runs**, not on its own. The run ends by printing `/health/`, which
|
||||||
|
includes the deployed version.
|
||||||
|
|
||||||
|
`up --wait` only returns once every container's healthcheck passes. `web`'s is
|
||||||
|
`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means
|
||||||
|
the app actually serves.
|
||||||
|
|
||||||
|
## Networking: why `web` is on the host network
|
||||||
|
|
||||||
|
Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT`
|
||||||
|
rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well
|
||||||
|
as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids
|
||||||
|
that, but at boot Docker races tailscaled for the address and loses.
|
||||||
|
|
||||||
|
So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host
|
||||||
|
port:
|
||||||
|
|
||||||
|
- **LAN:** UFW default-deny blocks it.
|
||||||
|
- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW
|
||||||
|
(checked on mempool-box), so the headscale ACL is the real control. It grants
|
||||||
|
`tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`.
|
||||||
|
- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches
|
||||||
|
them from the host network.
|
||||||
|
|
||||||
|
This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto`
|
||||||
|
(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only
|
||||||
|
thing that can reach gunicorn. Caddy overwrites whatever value the client sent.
|
||||||
|
|
||||||
|
## Secrets and seed users
|
||||||
|
|
||||||
|
The Django secret key, the Postgres password, the S3 secret and the three seed
|
||||||
|
passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They
|
||||||
|
reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose
|
||||||
|
interpolates. `docker-compose.yml` itself has no secrets.
|
||||||
|
|
||||||
|
The seed users are alice, bob and admin (superuser), with the fixture's usernames
|
||||||
|
and **vault** passwords. The fixture's `*-test-pass` passwords are published in
|
||||||
|
the app repo and this site is public. `seed_users` runs on every deploy and
|
||||||
|
resets each listed user's password, so rotating one means editing the vault and
|
||||||
|
re-running the playbook.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
ansible-vault view host_vars/jar_lid_staging_local/vault.yml # from ansible/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backups
|
||||||
|
|
||||||
|
`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses
|
||||||
|
`backup_source`. At 02:20 it stops `web` only, and
|
||||||
|
`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with:
|
||||||
|
|
||||||
|
- `db.dump`: `pg_dump -Fc`
|
||||||
|
- `seaweedfs/`: the photo store's data directory, as files
|
||||||
|
|
||||||
|
`web` is the only writer to the database and the photo store. With it stopped,
|
||||||
|
both are quiet and are captured at the same instant. `storage` is deliberately
|
||||||
|
left running: on this disk it can take minutes to restart, and the first test run
|
||||||
|
that stopped it left the app down for 26 minutes. Uploads fail while `web` is
|
||||||
|
down.
|
||||||
|
|
||||||
|
The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally,
|
||||||
|
and small-backups-box keeps three as well. Staging data is disposable.
|
||||||
|
|
||||||
|
### Restore
|
||||||
|
|
||||||
|
```sh
|
||||||
|
age -d -i ~/.age/counterweight_age jar-lid-archive-staging_<ts>.tar.age | tar -xf - -C /tmp/restore
|
||||||
|
# on the box, stack down:
|
||||||
|
docker compose -f /opt/jar-lid-archive/docker-compose.yml down
|
||||||
|
rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/
|
||||||
|
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db
|
||||||
|
docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \
|
||||||
|
pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump
|
||||||
|
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait
|
||||||
|
```
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
| What | Where |
|
||||||
|
|---|---|
|
||||||
|
| Host liveness and disk | infra/400 (automatic for `managed`) |
|
||||||
|
| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) |
|
||||||
|
| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) |
|
||||||
|
| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` |
|
||||||
|
|
||||||
|
infra/401 has nothing to watch here, because no systemd units of ours run on
|
||||||
|
this box.
|
||||||
|
|
||||||
|
## Size and disk
|
||||||
|
|
||||||
|
The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's
|
||||||
|
random reads keep that mirror close to saturated, and this box has measured
|
||||||
|
**~96% "full" I/O pressure** with nothing of its own running. Everything is slow
|
||||||
|
here, but especially container starts. The healthcheck `start_period`s are sized
|
||||||
|
for that. Moving the disk to `local-zfs` (NVMe) would fix it.
|
||||||
|
|
||||||
|
|
||||||
|
1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian,
|
||||||
|
the images, the photos and three days of local backups, and each backup is
|
||||||
|
about the size of the whole photo store. infra/400's disk check alarms at 85%.
|
||||||
57
ansible/roles/jar_lid_archive/defaults/main.yml
Normal file
57
ansible/roles/jar_lid_archive/defaults/main.yml
Normal file
|
|
@ -0,0 +1,57 @@
|
||||||
|
---
|
||||||
|
# Jar Lid Archive: Django (gunicorn) + Postgres + SeaweedFS (S3) as one compose
|
||||||
|
# stack. Source and CI live in forgejo.contrapeso.xyz/counterweight/jar-lid-archive.
|
||||||
|
#
|
||||||
|
# Per-deployment values (domain, port, retention) live in the host's host_vars,
|
||||||
|
# because the Caddy play on the edge host and the backup plays read them too and
|
||||||
|
# a role default cannot reach a second play.
|
||||||
|
|
||||||
|
jar_lid_archive_dir: /opt/jar-lid-archive
|
||||||
|
|
||||||
|
# ── Images ───────────────────────────────────────────────────────────────────
|
||||||
|
# The app follows a moving tag on purpose: every run pulls it, and compose
|
||||||
|
# recreates `web` only when the digest actually changed. CI pushes :master on
|
||||||
|
# every green build of master, plus :<sha>.
|
||||||
|
jar_lid_archive_image: forgejo.contrapeso.xyz/counterweight/jar-lid-archive
|
||||||
|
jar_lid_archive_image_tag: master
|
||||||
|
|
||||||
|
# Pinned to what the app's own compose.yaml and CI run, so staging rehearses the
|
||||||
|
# same stack. Bump them together with the app repo, not separately.
|
||||||
|
jar_lid_archive_postgres_image: "postgres:18"
|
||||||
|
jar_lid_archive_seaweedfs_image: "chrislusf/seaweedfs:4.47"
|
||||||
|
|
||||||
|
# Registry login. Empty = pull anonymously, which works today: the package is
|
||||||
|
# owner-scoped in Forgejo and public even though the repository is private.
|
||||||
|
jar_lid_archive_registry: forgejo.contrapeso.xyz
|
||||||
|
jar_lid_archive_registry_user: ""
|
||||||
|
jar_lid_archive_registry_token: ""
|
||||||
|
|
||||||
|
# ── Networking ───────────────────────────────────────────────────────────────
|
||||||
|
# gunicorn listens on the HOST network on this port. See the README for why host
|
||||||
|
# networking rather than a published port: Docker's published ports bypass UFW.
|
||||||
|
jar_lid_archive_port: 8000
|
||||||
|
# Loopback-only ports for the two backing services, so the web container (on the
|
||||||
|
# host network) can reach them and nothing else can.
|
||||||
|
jar_lid_archive_postgres_port: 5432
|
||||||
|
jar_lid_archive_s3_port: 8333
|
||||||
|
|
||||||
|
jar_lid_archive_domain: "" # e.g. jars-staging.contrapeso.xyz
|
||||||
|
|
||||||
|
# ── App settings ─────────────────────────────────────────────────────────────
|
||||||
|
jar_lid_archive_postgres_db: jarlid
|
||||||
|
jar_lid_archive_postgres_user: jarlid
|
||||||
|
jar_lid_archive_s3_access_key: jarlid
|
||||||
|
jar_lid_archive_s3_bucket: photos
|
||||||
|
|
||||||
|
# Secrets: pass from the vault. The assert in tasks/deploy.yml refuses empties.
|
||||||
|
jar_lid_archive_django_secret_key: ""
|
||||||
|
jar_lid_archive_postgres_password: ""
|
||||||
|
jar_lid_archive_s3_secret_key: ""
|
||||||
|
|
||||||
|
# ── Seed users ───────────────────────────────────────────────────────────────
|
||||||
|
# Loaded with `manage.py seed_users -` on every run. That command RESETS each
|
||||||
|
# listed user's password and flags, so the vault stays the source of truth.
|
||||||
|
# Same usernames as the app's tests/fixtures/users.json, never its passwords:
|
||||||
|
# this deployment is reachable from the internet.
|
||||||
|
# - {username: alice, password: "...", email: "...", is_superuser: false}
|
||||||
|
jar_lid_archive_seed_users: []
|
||||||
100
ansible/roles/jar_lid_archive/tasks/deploy.yml
Normal file
100
ansible/roles/jar_lid_archive/tasks/deploy.yml
Normal file
|
|
@ -0,0 +1,100 @@
|
||||||
|
---
|
||||||
|
- name: Assert the deployment is fully specified
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- jar_lid_archive_domain | length > 0
|
||||||
|
- jar_lid_archive_django_secret_key | length >= 50
|
||||||
|
- jar_lid_archive_postgres_password | length > 0
|
||||||
|
- jar_lid_archive_s3_secret_key | length > 0
|
||||||
|
fail_msg: >-
|
||||||
|
jar_lid_archive needs a domain and its three secrets from the vault. The
|
||||||
|
app falls back to development defaults for every one of them, so an empty
|
||||||
|
value here would not fail - it would deploy with a published secret key.
|
||||||
|
no_log: true
|
||||||
|
|
||||||
|
- name: Create the deployment directory
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ jar_lid_archive_dir }}"
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
# Existence only, no owner: the postgres and seaweedfs entrypoints chown their
|
||||||
|
# own data directories to the uid inside the container. Declaring an owner here
|
||||||
|
# would fight them on every run - the mempool MariaDB lesson.
|
||||||
|
- name: Ensure the data directories exist
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ jar_lid_archive_dir }}/{{ item }}"
|
||||||
|
state: directory
|
||||||
|
loop:
|
||||||
|
- postgres
|
||||||
|
- seaweedfs
|
||||||
|
|
||||||
|
# diff: false, because --diff prints rendered content and this is all secrets.
|
||||||
|
- name: Write the compose environment file
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: env.j2
|
||||||
|
dest: "{{ jar_lid_archive_dir }}/.env"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0600'
|
||||||
|
diff: false
|
||||||
|
|
||||||
|
- name: Write docker-compose.yml
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: docker-compose.yml.j2
|
||||||
|
dest: "{{ jar_lid_archive_dir }}/docker-compose.yml"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0644'
|
||||||
|
|
||||||
|
- name: Log in to the container registry
|
||||||
|
ansible.builtin.command:
|
||||||
|
cmd: "docker login {{ jar_lid_archive_registry }} -u {{ jar_lid_archive_registry_user }} --password-stdin"
|
||||||
|
stdin: "{{ jar_lid_archive_registry_token }}"
|
||||||
|
when: jar_lid_archive_registry_token | length > 0
|
||||||
|
changed_when: false
|
||||||
|
no_log: true
|
||||||
|
|
||||||
|
# Pulled on every run: staging follows :master. `up` below recreates `web` only
|
||||||
|
# when the pulled digest differs from the running one.
|
||||||
|
- name: Pull images
|
||||||
|
ansible.builtin.command: docker compose pull --quiet
|
||||||
|
args:
|
||||||
|
chdir: "{{ jar_lid_archive_dir }}"
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
# --wait blocks until every service's healthcheck passes, and `web`'s is
|
||||||
|
# /health/, which itself checks the database and the bucket - so this task
|
||||||
|
# succeeding means the app actually serves, not merely that containers exist.
|
||||||
|
- name: Start the stack and wait for it to be healthy
|
||||||
|
ansible.builtin.command: docker compose up -d --remove-orphans --wait --wait-timeout 300
|
||||||
|
args:
|
||||||
|
chdir: "{{ jar_lid_archive_dir }}"
|
||||||
|
register: jar_lid_archive_up
|
||||||
|
changed_when: jar_lid_archive_up.stderr is search('Created|Recreated|Started|Removed')
|
||||||
|
|
||||||
|
- name: Read the deployed version
|
||||||
|
ansible.builtin.uri:
|
||||||
|
url: "http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
|
||||||
|
return_content: true
|
||||||
|
register: jar_lid_archive_health
|
||||||
|
# Under --check the stack was never started, so there is nothing to ask.
|
||||||
|
when: not ansible_check_mode
|
||||||
|
|
||||||
|
- name: Show the deployed version
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg: "{{ jar_lid_archive_health.json }}"
|
||||||
|
when: not ansible_check_mode
|
||||||
|
|
||||||
|
# Used by the backup (services/jar-lid-archive/setup_backup_jar_lid_archive.yml).
|
||||||
|
# Lives here because what to dump is a fact about this stack's layout.
|
||||||
|
- name: Install the dump script
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: dump.sh.j2
|
||||||
|
dest: /usr/local/bin/jar-lid-archive-dump.sh
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0755'
|
||||||
|
validate: "bash -n %s"
|
||||||
74
ansible/roles/jar_lid_archive/tasks/docker.yml
Normal file
74
ansible/roles/jar_lid_archive/tasks/docker.yml
Normal file
|
|
@ -0,0 +1,74 @@
|
||||||
|
---
|
||||||
|
# Same Docker CE install as roles/mempool/tasks/docker.yml. infra/910_docker
|
||||||
|
# targets every managed host, so each Docker role installs its own engine.
|
||||||
|
- name: Remove old Docker-related packages
|
||||||
|
apt:
|
||||||
|
name:
|
||||||
|
- docker.io
|
||||||
|
- docker-doc
|
||||||
|
- docker-compose
|
||||||
|
- podman-docker
|
||||||
|
- containerd
|
||||||
|
- runc
|
||||||
|
state: absent
|
||||||
|
purge: yes
|
||||||
|
autoremove: yes
|
||||||
|
|
||||||
|
- name: Update apt cache
|
||||||
|
apt:
|
||||||
|
update_cache: yes
|
||||||
|
|
||||||
|
- name: Install prerequisites
|
||||||
|
apt:
|
||||||
|
name:
|
||||||
|
- ca-certificates
|
||||||
|
- curl
|
||||||
|
state: present
|
||||||
|
|
||||||
|
- name: Create directory for Docker GPG key
|
||||||
|
file:
|
||||||
|
path: /etc/apt/keyrings
|
||||||
|
state: directory
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
- name: Download Docker GPG key
|
||||||
|
get_url:
|
||||||
|
url: https://download.docker.com/linux/debian/gpg
|
||||||
|
dest: /etc/apt/keyrings/docker.asc
|
||||||
|
mode: '0644'
|
||||||
|
|
||||||
|
- name: Get Debian architecture
|
||||||
|
command: dpkg --print-architecture
|
||||||
|
register: deb_arch
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Add Docker repository
|
||||||
|
apt_repository:
|
||||||
|
repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable"
|
||||||
|
filename: docker
|
||||||
|
state: present
|
||||||
|
update_cache: yes
|
||||||
|
|
||||||
|
- name: Install Docker packages
|
||||||
|
apt:
|
||||||
|
name:
|
||||||
|
- docker-ce
|
||||||
|
- docker-ce-cli
|
||||||
|
- containerd.io
|
||||||
|
- docker-buildx-plugin
|
||||||
|
- docker-compose-plugin
|
||||||
|
state: present
|
||||||
|
update_cache: yes
|
||||||
|
|
||||||
|
- name: Ensure Docker is started and enabled
|
||||||
|
systemd:
|
||||||
|
name: docker
|
||||||
|
enabled: yes
|
||||||
|
state: started
|
||||||
|
|
||||||
|
- name: Add user to docker group
|
||||||
|
user:
|
||||||
|
name: "{{ ansible_user }}"
|
||||||
|
groups: docker
|
||||||
|
append: yes
|
||||||
|
|
||||||
6
ansible/roles/jar_lid_archive/tasks/main.yml
Normal file
6
ansible/roles/jar_lid_archive/tasks/main.yml
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
---
|
||||||
|
# import_tasks, not include_tasks: static imports stay visible to --list-tasks,
|
||||||
|
# which is the main verification tool in this repo.
|
||||||
|
- ansible.builtin.import_tasks: docker.yml
|
||||||
|
- ansible.builtin.import_tasks: deploy.yml
|
||||||
|
- ansible.builtin.import_tasks: seed.yml
|
||||||
13
ansible/roles/jar_lid_archive/tasks/seed.yml
Normal file
13
ansible/roles/jar_lid_archive/tasks/seed.yml
Normal file
|
|
@ -0,0 +1,13 @@
|
||||||
|
---
|
||||||
|
# seed_users is idempotent but RESETS each listed user's password and flags, so
|
||||||
|
# re-running it is how a rotated vault password reaches the app. It reports
|
||||||
|
# "N created, M updated"; only a creation counts as a change.
|
||||||
|
- name: Seed users
|
||||||
|
ansible.builtin.command:
|
||||||
|
cmd: docker compose exec -T web python manage.py seed_users -
|
||||||
|
chdir: "{{ jar_lid_archive_dir }}"
|
||||||
|
stdin: "{{ jar_lid_archive_seed_users | to_json }}"
|
||||||
|
register: jar_lid_archive_seed
|
||||||
|
changed_when: "jar_lid_archive_seed.stdout is not search('Seeded users: 0 created')"
|
||||||
|
when: jar_lid_archive_seed_users | length > 0
|
||||||
|
no_log: true
|
||||||
123
ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2
Normal file
123
ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
|
||||||
|
#
|
||||||
|
# Same three services as the app repo's compose.yaml, with three differences:
|
||||||
|
# - `web` runs the CI-built image instead of `build: .`
|
||||||
|
# - `web` is on the HOST network, so UFW and the tailnet ACL decide who reaches
|
||||||
|
# gunicorn. A published port would be DNATed by Docker straight past UFW and
|
||||||
|
# answer on the LAN too. See the role README.
|
||||||
|
# - data is bind-mounted under {{ jar_lid_archive_dir }} rather than in named
|
||||||
|
# volumes, so the backup tars a path it can name.
|
||||||
|
#
|
||||||
|
# Secrets are interpolated from .env (0600) beside this file; this file has none.
|
||||||
|
name: jar-lid-archive
|
||||||
|
|
||||||
|
x-logging: &logging
|
||||||
|
# Docker's default json-file driver never rotates, on a 10 GB disk.
|
||||||
|
driver: json-file
|
||||||
|
options:
|
||||||
|
max-size: "10m"
|
||||||
|
max-file: "3"
|
||||||
|
|
||||||
|
x-postgres-env: &postgres-env
|
||||||
|
POSTGRES_DB: "{{ jar_lid_archive_postgres_db }}"
|
||||||
|
POSTGRES_USER: "{{ jar_lid_archive_postgres_user }}"
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?}
|
||||||
|
|
||||||
|
x-s3-env: &s3-env
|
||||||
|
S3_ACCESS_KEY: "{{ jar_lid_archive_s3_access_key }}"
|
||||||
|
S3_SECRET_KEY: ${S3_SECRET_KEY:?}
|
||||||
|
|
||||||
|
services:
|
||||||
|
web:
|
||||||
|
image: "{{ jar_lid_archive_image }}:{{ jar_lid_archive_image_tag }}"
|
||||||
|
# gunicorn binds 0.0.0.0:{{ jar_lid_archive_port }} - the port is fixed by the image's CMD.
|
||||||
|
network_mode: host
|
||||||
|
environment:
|
||||||
|
<<: [*postgres-env, *s3-env]
|
||||||
|
POSTGRES_HOST: 127.0.0.1
|
||||||
|
POSTGRES_PORT: "{{ jar_lid_archive_postgres_port }}"
|
||||||
|
S3_ENDPOINT_URL: "http://127.0.0.1:{{ jar_lid_archive_s3_port }}"
|
||||||
|
S3_BUCKET: "{{ jar_lid_archive_s3_bucket }}"
|
||||||
|
DJANGO_DEBUG: "0"
|
||||||
|
DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY:?}
|
||||||
|
# 127.0.0.1 for the container healthcheck and the probe, which both call
|
||||||
|
# /health/ directly rather than through Caddy.
|
||||||
|
DJANGO_ALLOWED_HOSTS: "{{ jar_lid_archive_domain }},127.0.0.1,localhost"
|
||||||
|
# Caddy terminates TLS and forwards plain HTTP with X-Forwarded-Proto. The
|
||||||
|
# app must trust that header or every POST fails CSRF and share links come
|
||||||
|
# out http://. Safe only because nothing but Caddy can reach gunicorn.
|
||||||
|
DJANGO_BEHIND_TLS_PROXY: "1"
|
||||||
|
DJANGO_CSRF_TRUSTED_ORIGINS: "https://{{ jar_lid_archive_domain }}"
|
||||||
|
depends_on:
|
||||||
|
db:
|
||||||
|
condition: service_healthy
|
||||||
|
storage:
|
||||||
|
condition: service_healthy
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:{{ jar_lid_archive_port }}/health/')"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
start_period: 180s
|
||||||
|
logging: *logging
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
db:
|
||||||
|
image: "{{ jar_lid_archive_postgres_image }}"
|
||||||
|
environment: *postgres-env
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:{{ jar_lid_archive_postgres_port }}:5432"
|
||||||
|
volumes:
|
||||||
|
- ./postgres:/var/lib/postgresql
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 15
|
||||||
|
start_period: 120s
|
||||||
|
logging: *logging
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
storage:
|
||||||
|
image: "{{ jar_lid_archive_seaweedfs_image }}"
|
||||||
|
environment: *s3-env
|
||||||
|
# Copied from the app's compose.yaml: write the S3 identity from the
|
||||||
|
# environment, then hand over to the image's entrypoint, which drops root
|
||||||
|
# and runs master + volume + filer + S3 gateway.
|
||||||
|
#
|
||||||
|
# The `rm -f` is ours, and load-bearing. On a RESTART of the same container
|
||||||
|
# (a reboot, a backup's stop/start) /tmp/s3.json survives, owned by seaweed
|
||||||
|
# from the chown below. Root cannot then re-create it: /tmp is sticky and
|
||||||
|
# the kernel's protected_regular refuses O_CREAT on another user's file
|
||||||
|
# there. The result was "can't create /tmp/s3.json: Permission denied", an
|
||||||
|
# unhealthy storage container, and web never starting - found on the first
|
||||||
|
# backup run.
|
||||||
|
entrypoint: ["sh", "-c"]
|
||||||
|
command:
|
||||||
|
- |
|
||||||
|
rm -f /tmp/s3.json
|
||||||
|
umask 077
|
||||||
|
cat > /tmp/s3.json <<EOF
|
||||||
|
{"identities": [{"name": "app",
|
||||||
|
"credentials": [{"accessKey": "$$S3_ACCESS_KEY", "secretKey": "$$S3_SECRET_KEY"}],
|
||||||
|
"actions": ["Admin", "Read", "Write", "List", "Tagging"]}]}
|
||||||
|
EOF
|
||||||
|
chown seaweed /tmp/s3.json
|
||||||
|
exec /entrypoint.sh server -s3 -s3.config=/tmp/s3.json
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:{{ jar_lid_archive_s3_port }}:8333"
|
||||||
|
volumes:
|
||||||
|
- ./seaweedfs:/data
|
||||||
|
# start_period, because this VM's disk is on nodito's HDD mirror, shared
|
||||||
|
# with knots-box. Under that contention SeaweedFS has taken over the 150s
|
||||||
|
# that retries x interval allowed; compose then marked it unhealthy, and
|
||||||
|
# web's depends_on gave up and never started it - a 26-minute outage.
|
||||||
|
# Failures inside start_period do not count.
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8333/healthz"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 30
|
||||||
|
start_period: 300s
|
||||||
|
logging: *logging
|
||||||
|
restart: unless-stopped
|
||||||
25
ansible/roles/jar_lid_archive/templates/dump.sh.j2
Normal file
25
ansible/roles/jar_lid_archive/templates/dump.sh.j2
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Jar Lid Archive dump - managed by Ansible (roles/jar_lid_archive).
|
||||||
|
#
|
||||||
|
# Writes ONE uncompressed tar to STDOUT, for roles/backup_source to encrypt:
|
||||||
|
# db.dump pg_dump custom format -> pg_restore -d jarlid db.dump
|
||||||
|
# seaweedfs/ the photo store, as files
|
||||||
|
#
|
||||||
|
# Called with `web` stopped (the backup playbook arranges it). web is the only
|
||||||
|
# writer to db and storage, so both are quiescent and agree with each other: no
|
||||||
|
# row points at a photo the tar does not have. storage keeps running; with no
|
||||||
|
# writes arriving its volume files do not change under tar.
|
||||||
|
#
|
||||||
|
# The pg_dump lands in a temp dir on the same disk as the live database, which
|
||||||
|
# is already plaintext; it is gone before this script exits.
|
||||||
|
#
|
||||||
|
# tar, not tar.gz: the bulk is JPEGs, which do not compress, and gzip on one
|
||||||
|
# core would only make the window longer.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
cd {{ jar_lid_archive_dir }}
|
||||||
|
tmp=$(mktemp -d {{ jar_lid_archive_dir }}/.dump.XXXXXX)
|
||||||
|
trap 'rm -rf "$tmp"' EXIT
|
||||||
|
|
||||||
|
docker compose exec -T db pg_dump -U {{ jar_lid_archive_postgres_user }} -d {{ jar_lid_archive_postgres_db }} -Fc > "$tmp/db.dump"
|
||||||
|
tar -cf - -C "$tmp" db.dump -C {{ jar_lid_archive_dir }} seaweedfs
|
||||||
5
ansible/roles/jar_lid_archive/templates/env.j2
Normal file
5
ansible/roles/jar_lid_archive/templates/env.j2
Normal file
|
|
@ -0,0 +1,5 @@
|
||||||
|
# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
|
||||||
|
# Read by docker compose to interpolate docker-compose.yml. 0600: secrets.
|
||||||
|
POSTGRES_PASSWORD={{ jar_lid_archive_postgres_password }}
|
||||||
|
S3_SECRET_KEY={{ jar_lid_archive_s3_secret_key }}
|
||||||
|
DJANGO_SECRET_KEY={{ jar_lid_archive_django_secret_key }}
|
||||||
|
|
@ -43,6 +43,7 @@
|
||||||
"tag:mempool-box:8080",
|
"tag:mempool-box:8080",
|
||||||
"tag:memos-box:5230",
|
"tag:memos-box:5230",
|
||||||
"tag:music-box:4533",
|
"tag:music-box:4533",
|
||||||
|
"tag:jar-lid-archive-staging-box:8000",
|
||||||
"tag:monerod-box:18089",
|
"tag:monerod-box:18089",
|
||||||
"tag:monerod-box:8081",
|
"tag:monerod-box:8081",
|
||||||
"tag:arbret-staging-box:80",
|
"tag:arbret-staging-box:80",
|
||||||
|
|
@ -82,7 +83,8 @@
|
||||||
"tag:prd-arbret:22",
|
"tag:prd-arbret:22",
|
||||||
"tag:prd-vipy:22",
|
"tag:prd-vipy:22",
|
||||||
"tag:memos-box:22",
|
"tag:memos-box:22",
|
||||||
"tag:stalwart-box:22"
|
"tag:stalwart-box:22",
|
||||||
|
"tag:jar-lid-archive-staging-box:22"
|
||||||
],
|
],
|
||||||
"src": [
|
"src": [
|
||||||
"tag:small-backups-box"
|
"tag:small-backups-box"
|
||||||
|
|
@ -102,6 +104,9 @@
|
||||||
"tag:fulcrum-box": [
|
"tag:fulcrum-box": [
|
||||||
"{{ headscale_namespace }}@"
|
"{{ headscale_namespace }}@"
|
||||||
],
|
],
|
||||||
|
"tag:jar-lid-archive-staging-box": [
|
||||||
|
"{{ headscale_namespace }}@"
|
||||||
|
],
|
||||||
"tag:knots-box": [
|
"tag:knots-box": [
|
||||||
"{{ headscale_namespace }}@"
|
"{{ headscale_namespace }}@"
|
||||||
],
|
],
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,51 @@
|
||||||
|
---
|
||||||
|
# Jar Lid Archive on its own box, published from the edge host.
|
||||||
|
#
|
||||||
|
# See roles/jar_lid_archive/README.md. Per-host values (domain, port, seed users,
|
||||||
|
# secrets) are in host_vars/<host>/, so a production host is a new inventory
|
||||||
|
# entry in [jar_lid_archive], not a new playbook.
|
||||||
|
- name: Deploy Jar Lid Archive
|
||||||
|
hosts: jar_lid_archive
|
||||||
|
become: yes
|
||||||
|
roles:
|
||||||
|
- jar_lid_archive
|
||||||
|
# "Does the app answer", as opposed to "are the containers up". /health/
|
||||||
|
# runs SELECT 1 and touches the bucket, and returns 503 if either fails.
|
||||||
|
- role: healthcheck
|
||||||
|
vars:
|
||||||
|
healthcheck_name: jar-lid-archive
|
||||||
|
healthcheck_description: "Jar Lid Archive /health/ (app, database and photo store)"
|
||||||
|
healthcheck_command: "curl -fsS --max-time 10 -o /dev/null http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
|
||||||
|
healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_{{ jar_lid_archive_instance }}/external"
|
||||||
|
healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
|
||||||
|
|
||||||
|
- name: Publish Jar Lid Archive from the edge host
|
||||||
|
hosts: edge
|
||||||
|
become: yes
|
||||||
|
tasks:
|
||||||
|
# No basic auth: the one-off links (/links/...) are opened by people who have
|
||||||
|
# no account and no proxy credential. The app's own login guards the rest.
|
||||||
|
#
|
||||||
|
# Caddy sets X-Forwarded-Proto: https and overwrites any value the client
|
||||||
|
# sent, which is what lets the app trust it (DJANGO_BEHIND_TLS_PROXY).
|
||||||
|
#
|
||||||
|
# No request size limit is set, deliberately: Caddy has none by default, and
|
||||||
|
# phones upload multi-megabyte photos.
|
||||||
|
- name: Expose Jar Lid Archive over HTTPS
|
||||||
|
ansible.builtin.include_role:
|
||||||
|
name: caddy_site
|
||||||
|
vars:
|
||||||
|
caddy_site_name: "{{ hostvars[jla_host].jar_lid_archive_instance }}"
|
||||||
|
caddy_site_domain: "{{ hostvars[jla_host].jar_lid_archive_domain }}"
|
||||||
|
# Tailscale MagicDNS: the box has no public address.
|
||||||
|
caddy_site_upstream: "{{ hostvars[jla_host].ansible_host }}:{{ hostvars[jla_host].jar_lid_archive_port }}"
|
||||||
|
caddy_site_resolvers: "100.100.100.100"
|
||||||
|
loop: "{{ groups['jar_lid_archive'] }}"
|
||||||
|
# Not `item`: the role's own tasks loop, and would shadow it.
|
||||||
|
loop_control:
|
||||||
|
loop_var: jla_host
|
||||||
|
|
||||||
|
# The per-site validate cannot catch a conflict between sites.
|
||||||
|
- name: Validate the whole Caddyfile
|
||||||
|
ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
|
||||||
|
changed_when: false
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
---
|
||||||
|
# Jar Lid Archive backup: the database and the photo store, as ONE artefact,
|
||||||
|
# dumped and age-encrypted on the box itself. small-backups-box pulls it.
|
||||||
|
#
|
||||||
|
# Only `web` is stopped. It is the only writer to both the database and the
|
||||||
|
# photo store, so with it down both are quiescent and are captured at the same
|
||||||
|
# instant - no row points at a photo the tar lacks. db stays up for pg_dump.
|
||||||
|
#
|
||||||
|
# storage is deliberately NOT stopped, though an earlier version did: on this
|
||||||
|
# VM's contended HDD-backed disk SeaweedFS can take minutes to come back, and the
|
||||||
|
# first test run left the app down for 26 minutes waiting on it.
|
||||||
|
- name: Configure the Jar Lid Archive backup on its own host
|
||||||
|
hosts: jar_lid_archive
|
||||||
|
become: yes
|
||||||
|
|
||||||
|
tasks:
|
||||||
|
- name: Ensure Jar Lid Archive dumps itself, encrypted, on a timer
|
||||||
|
ansible.builtin.include_role:
|
||||||
|
name: backup_source
|
||||||
|
vars:
|
||||||
|
backup_source_name: "{{ jar_lid_archive_instance }}"
|
||||||
|
backup_source_description: "Jar Lid Archive ({{ jar_lid_archive_instance }})"
|
||||||
|
backup_source_dump_command: /usr/local/bin/jar-lid-archive-dump.sh
|
||||||
|
# Uncompressed: the bulk is JPEGs. See roles/jar_lid_archive/templates/dump.sh.j2.
|
||||||
|
backup_source_artifact_suffix: "tar.age"
|
||||||
|
backup_source_stop_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml stop web"
|
||||||
|
# `up -d --wait`, not `start`: it also repairs a db or storage that died
|
||||||
|
# meanwhile, and only returns once /health/ passes.
|
||||||
|
backup_source_start_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait --wait-timeout 300"
|
||||||
|
backup_source_retention_days: "{{ jar_lid_archive_backup_retention_days }}"
|
||||||
|
backup_source_on_calendar: "*-*-* 02:20:00"
|
||||||
|
# Reported to Gatus as backup-dump_<instance>.
|
||||||
|
backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_{{ jar_lid_archive_instance }}/external"
|
||||||
|
backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
|
||||||
|
|
@ -57,6 +57,7 @@
|
||||||
- import_playbook: services/xmrblocks/deploy_xmrblocks_playbook.yml
|
- import_playbook: services/xmrblocks/deploy_xmrblocks_playbook.yml
|
||||||
- import_playbook: services/memos/deploy_memos_playbook.yml
|
- import_playbook: services/memos/deploy_memos_playbook.yml
|
||||||
- import_playbook: services/navidrome/deploy_navidrome_playbook.yml
|
- import_playbook: services/navidrome/deploy_navidrome_playbook.yml
|
||||||
|
- import_playbook: services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml
|
||||||
- import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml
|
- import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml
|
||||||
- import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml
|
- import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml
|
||||||
- import_playbook: services/headscale/deploy_headscale_playbook.yml
|
- import_playbook: services/headscale/deploy_headscale_playbook.yml
|
||||||
|
|
@ -72,6 +73,7 @@
|
||||||
- import_playbook: services/forgejo/setup_backup_forgejo.yml
|
- import_playbook: services/forgejo/setup_backup_forgejo.yml
|
||||||
- import_playbook: services/lnbits/setup_backup_lnbits.yml
|
- import_playbook: services/lnbits/setup_backup_lnbits.yml
|
||||||
- import_playbook: services/memos/setup_backup_memos.yml
|
- import_playbook: services/memos/setup_backup_memos.yml
|
||||||
|
- import_playbook: services/jar-lid-archive/setup_backup_jar_lid_archive.yml
|
||||||
- import_playbook: playbooks/backups.yml
|
- import_playbook: playbooks/backups.yml
|
||||||
|
|
||||||
# Deliberately not here. Every playbook in the repo is either imported above or
|
# Deliberately not here. Every playbook in the repo is either imported above or
|
||||||
|
|
|
||||||
|
|
@ -90,6 +90,14 @@ vms = {
|
||||||
disk_size_gb = 20
|
disk_size_gb = 20
|
||||||
ipconfig0 = "ip=dhcp"
|
ipconfig0 = "ip=dhcp"
|
||||||
},
|
},
|
||||||
|
"jar-lid-archive-staging-box" = {
|
||||||
|
vmid = 3300
|
||||||
|
name = "jar-lid-archive-staging-box"
|
||||||
|
cores = 1
|
||||||
|
memory_mb = 1024
|
||||||
|
disk_size_gb = 10
|
||||||
|
ipconfig0 = "ip=dhcp"
|
||||||
|
},
|
||||||
# disk_size_gb here is the music library's ceiling, not just the OS: the
|
# disk_size_gb here is the music library's ceiling, not just the OS: the
|
||||||
# library lives on the root disk at /srv/music. ~45 GB is usable after
|
# library lives on the root disk at /srv/music. ~45 GB is usable after
|
||||||
# Debian, so roughly 300-450 albums in FLAC. Growing it means a Proxmox
|
# Debian, so roughly 300-450 albums in FLAC. Growing it means a Proxmox
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue