jar-lid-archive: a staging deployment on its own box

Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ansible/roles/jar_lid_archive/defaults/main.yml

@@ -0,0 +1,57 @@
+---
+# Jar Lid Archive: Django (gunicorn) + Postgres + SeaweedFS (S3) as one compose
+# stack. Source and CI live in forgejo.contrapeso.xyz/counterweight/jar-lid-archive.
+#
+# Per-deployment values (domain, port, retention) live in the host's host_vars,
+# because the Caddy play on the edge host and the backup plays read them too and
+# a role default cannot reach a second play.
+
+jar_lid_archive_dir: /opt/jar-lid-archive
+
+# ── Images ───────────────────────────────────────────────────────────────────
+# The app follows a moving tag on purpose: every run pulls it, and compose
+# recreates `web` only when the digest actually changed. CI pushes :master on
+# every green build of master, plus :<sha>.
+jar_lid_archive_image: forgejo.contrapeso.xyz/counterweight/jar-lid-archive
+jar_lid_archive_image_tag: master
+
+# Pinned to what the app's own compose.yaml and CI run, so staging rehearses the
+# same stack. Bump them together with the app repo, not separately.
+jar_lid_archive_postgres_image: "postgres:18"
+jar_lid_archive_seaweedfs_image: "chrislusf/seaweedfs:4.47"
+
+# Registry login. Empty = pull anonymously, which works today: the package is
+# owner-scoped in Forgejo and public even though the repository is private.
+jar_lid_archive_registry: forgejo.contrapeso.xyz
+jar_lid_archive_registry_user: ""
+jar_lid_archive_registry_token: ""
+
+# ── Networking ───────────────────────────────────────────────────────────────
+# gunicorn listens on the HOST network on this port. See the README for why host
+# networking rather than a published port: Docker's published ports bypass UFW.
+jar_lid_archive_port: 8000
+# Loopback-only ports for the two backing services, so the web container (on the
+# host network) can reach them and nothing else can.
+jar_lid_archive_postgres_port: 5432
+jar_lid_archive_s3_port: 8333
+
+jar_lid_archive_domain: ""              # e.g. jars-staging.contrapeso.xyz
+
+# ── App settings ─────────────────────────────────────────────────────────────
+jar_lid_archive_postgres_db: jarlid
+jar_lid_archive_postgres_user: jarlid
+jar_lid_archive_s3_access_key: jarlid
+jar_lid_archive_s3_bucket: photos
+
+# Secrets: pass from the vault. The assert in tasks/deploy.yml refuses empties.
+jar_lid_archive_django_secret_key: ""
+jar_lid_archive_postgres_password: ""
+jar_lid_archive_s3_secret_key: ""
+
+# ── Seed users ───────────────────────────────────────────────────────────────
+# Loaded with `manage.py seed_users -` on every run. That command RESETS each
+# listed user's password and flags, so the vault stays the source of truth.
+# Same usernames as the app's tests/fixtures/users.json, never its passwords:
+# this deployment is reachable from the internet.
+#   - {username: alice, password: "...", email: "...", is_superuser: false}
+jar_lid_archive_seed_users: []