jar-lid-archive: a staging deployment on its own box

Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ansible/host_vars/jar_lid_staging_local/main.yml

@@ -0,0 +1,28 @@
+---
+# Jar Lid Archive STAGING. Secrets are in vault.yml beside this file.
+#
+# These live here rather than in the role defaults because three plays read
+# them: the deploy on this box, the Caddy vhost on the edge host (port, domain)
+# and the backup (retention). A role default cannot reach a second play.
+
+# Names this deployment everywhere it must be unique across hosts: the Caddy site
+# file, the Gatus probe (probe_<instance>), the backup artefacts and the
+# backup-dump/backup-store endpoints.
+jar_lid_archive_instance: jar-lid-archive-staging
+
+# Fixed by the image's CMD (gunicorn --bind 0.0.0.0:8000); changing this moves
+# what Caddy and the probe call, not where gunicorn listens.
+jar_lid_archive_port: 8000
+jar_lid_archive_domain: "{{ subdomains.jar_lid_archive_staging }}.{{ root_domain }}"
+
+# Staging data is disposable: three days locally and on small-backups-box.
+jar_lid_archive_backup_retention_days: 3
+
+jar_lid_archive_seed_users:
+  - {username: alice, email: alice@example.com, password: "{{ jar_lid_archive_seed_password_alice }}"}
+  - {username: bob,   email: bob@example.com,   password: "{{ jar_lid_archive_seed_password_bob }}"}
+  - {username: admin, email: admin@example.com, password: "{{ jar_lid_archive_seed_password_admin }}", is_superuser: true}
+
+# No systemd units of ours run here - the stack is docker compose - so infra/401
+# has nothing to watch. The probe in the deploy playbook calls /health/, which
+# checks the database and the bucket as well as gunicorn.

ansible/host_vars/jar_lid_staging_local/vault.yml

@@ -0,0 +1,34 @@
+$ANSIBLE_VAULT;1.1;AES256
+37383535306133363031316535656461613038653562373839363432623163356234303061643566
+6533333535613064653336313664643063326562666639640a373530386539316363346332663137
+31303230323065623733663666383039346337343239646465393166663239643836356639373765
+3564313065363266360a643239333330616463616533623435656164386264303634356134303831
+36356131323637363734316161383065653663623732386638363165633265356362386538643039
+64636464323730316638316430643339356662353736366332663738656566663137366166393735
+34633430363931323330393234336661383532653366633735313365363562396137356438353330
+65313737313131326361326631363864396536343633623963303632623465363964373964393430
+61323161663939346435386435616364653336646165356232346335353235356339656336306138
+65613636353636623562613362633761333637613066663838396638613161663532393030303035
+35316464356539666631636265646433353537396564366538643962373364383730626164316265
+63393034663835346362636564316135326430633963393139646365333666663630336462323361
+31353830633335366635366435643864313665366563396134313938666537663232343162383366
+34636231636261663739383366363962366438323962393732393864623238343961363035333866
+61386630396334626564306536336561333265383464623730386263373132613438396332356631
+30386435333739653637633032313966626437303634613838653735373666343565306663376230
+61303734623865353835626464613033346437623234626136353664303434386262346634356463
+34396433353431393935386136333331643539633138636135353339323930386634393639666665
+36643462653061363062383039313936303062393364353366623965313530303031306336396336
+65643663343163636338393666386134366337633064376530303463393737396165663930383562
+64316363663636383764656139366433393766613262613433623637393538633130616232373136
+31376636633666626365376533356262393065656539613266326465643663383439373661393631
+30636136363066653936653837376634356333353036633165316662383337636435323162316430
+30316331353361396132653233623034376539323064363135306262376638353664643163653334
+37643638323038663531376336313064383238366430663363353733356530393661646138303230
+30633862626166653362346632626261636531373130353034656532386230616330643134383631
+63396138326336626431656130306531626539323934386332306361396239376466623866613364
+36666632316331326361373136326465323365653034363337313135656262623465306439646362
+30636639663163643430363165316231366232623332393635623461306339313762323265623732
+30633464343235323566356434376535313238663533633630356163336532303765633366316166
+38633462303762323930393337306265393033653061616337646163333335333737666332666632
+35613132646636333335313263343136323337616662623465653939336637653134383766336265
+35633638663636353537333238643236623033636163333164663062353863643164