jar-lid-archive: a staging deployment on its own box

Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2

@@ -0,0 +1,123 @@
+# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
+#
+# Same three services as the app repo's compose.yaml, with three differences:
+#   - `web` runs the CI-built image instead of `build: .`
+#   - `web` is on the HOST network, so UFW and the tailnet ACL decide who reaches
+#     gunicorn. A published port would be DNATed by Docker straight past UFW and
+#     answer on the LAN too. See the role README.
+#   - data is bind-mounted under {{ jar_lid_archive_dir }} rather than in named
+#     volumes, so the backup tars a path it can name.
+#
+# Secrets are interpolated from .env (0600) beside this file; this file has none.
+name: jar-lid-archive
+
+x-logging: &logging
+  # Docker's default json-file driver never rotates, on a 10 GB disk.
+  driver: json-file
+  options:
+    max-size: "10m"
+    max-file: "3"
+
+x-postgres-env: &postgres-env
+  POSTGRES_DB: "{{ jar_lid_archive_postgres_db }}"
+  POSTGRES_USER: "{{ jar_lid_archive_postgres_user }}"
+  POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?}
+
+x-s3-env: &s3-env
+  S3_ACCESS_KEY: "{{ jar_lid_archive_s3_access_key }}"
+  S3_SECRET_KEY: ${S3_SECRET_KEY:?}
+
+services:
+  web:
+    image: "{{ jar_lid_archive_image }}:{{ jar_lid_archive_image_tag }}"
+    # gunicorn binds 0.0.0.0:{{ jar_lid_archive_port }} - the port is fixed by the image's CMD.
+    network_mode: host
+    environment:
+      <<: [*postgres-env, *s3-env]
+      POSTGRES_HOST: 127.0.0.1
+      POSTGRES_PORT: "{{ jar_lid_archive_postgres_port }}"
+      S3_ENDPOINT_URL: "http://127.0.0.1:{{ jar_lid_archive_s3_port }}"
+      S3_BUCKET: "{{ jar_lid_archive_s3_bucket }}"
+      DJANGO_DEBUG: "0"
+      DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY:?}
+      # 127.0.0.1 for the container healthcheck and the probe, which both call
+      # /health/ directly rather than through Caddy.
+      DJANGO_ALLOWED_HOSTS: "{{ jar_lid_archive_domain }},127.0.0.1,localhost"
+      # Caddy terminates TLS and forwards plain HTTP with X-Forwarded-Proto. The
+      # app must trust that header or every POST fails CSRF and share links come
+      # out http://. Safe only because nothing but Caddy can reach gunicorn.
+      DJANGO_BEHIND_TLS_PROXY: "1"
+      DJANGO_CSRF_TRUSTED_ORIGINS: "https://{{ jar_lid_archive_domain }}"
+    depends_on:
+      db:
+        condition: service_healthy
+      storage:
+        condition: service_healthy
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:{{ jar_lid_archive_port }}/health/')"]
+      interval: 30s
+      timeout: 5s
+      retries: 5
+      start_period: 180s
+    logging: *logging
+    restart: unless-stopped
+
+  db:
+    image: "{{ jar_lid_archive_postgres_image }}"
+    environment: *postgres-env
+    ports:
+      - "127.0.0.1:{{ jar_lid_archive_postgres_port }}:5432"
+    volumes:
+      - ./postgres:/var/lib/postgresql
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
+      interval: 5s
+      timeout: 3s
+      retries: 15
+      start_period: 120s
+    logging: *logging
+    restart: unless-stopped
+
+  storage:
+    image: "{{ jar_lid_archive_seaweedfs_image }}"
+    environment: *s3-env
+    # Copied from the app's compose.yaml: write the S3 identity from the
+    # environment, then hand over to the image's entrypoint, which drops root
+    # and runs master + volume + filer + S3 gateway.
+    #
+    # The `rm -f` is ours, and load-bearing. On a RESTART of the same container
+    # (a reboot, a backup's stop/start) /tmp/s3.json survives, owned by seaweed
+    # from the chown below. Root cannot then re-create it: /tmp is sticky and
+    # the kernel's protected_regular refuses O_CREAT on another user's file
+    # there. The result was "can't create /tmp/s3.json: Permission denied", an
+    # unhealthy storage container, and web never starting - found on the first
+    # backup run.
+    entrypoint: ["sh", "-c"]
+    command:
+      - |
+        rm -f /tmp/s3.json
+        umask 077
+        cat > /tmp/s3.json <<EOF
+        {"identities": [{"name": "app",
+          "credentials": [{"accessKey": "$$S3_ACCESS_KEY", "secretKey": "$$S3_SECRET_KEY"}],
+          "actions": ["Admin", "Read", "Write", "List", "Tagging"]}]}
+        EOF
+        chown seaweed /tmp/s3.json
+        exec /entrypoint.sh server -s3 -s3.config=/tmp/s3.json
+    ports:
+      - "127.0.0.1:{{ jar_lid_archive_s3_port }}:8333"
+    volumes:
+      - ./seaweedfs:/data
+    # start_period, because this VM's disk is on nodito's HDD mirror, shared
+    # with knots-box. Under that contention SeaweedFS has taken over the 150s
+    # that retries x interval allowed; compose then marked it unhealthy, and
+    # web's depends_on gave up and never started it - a 26-minute outage.
+    # Failures inside start_period do not count.
+    healthcheck:
+      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8333/healthz"]
+      interval: 5s
+      timeout: 3s
+      retries: 30
+      start_period: 300s
+    logging: *logging
+    restart: unless-stopped

ansible/roles/jar_lid_archive/templates/dump.sh.j2

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Jar Lid Archive dump - managed by Ansible (roles/jar_lid_archive).
+#
+# Writes ONE uncompressed tar to STDOUT, for roles/backup_source to encrypt:
+#   db.dump      pg_dump custom format   -> pg_restore -d jarlid db.dump
+#   seaweedfs/   the photo store, as files
+#
+# Called with `web` stopped (the backup playbook arranges it). web is the only
+# writer to db and storage, so both are quiescent and agree with each other: no
+# row points at a photo the tar does not have. storage keeps running; with no
+# writes arriving its volume files do not change under tar.
+#
+# The pg_dump lands in a temp dir on the same disk as the live database, which
+# is already plaintext; it is gone before this script exits.
+#
+# tar, not tar.gz: the bulk is JPEGs, which do not compress, and gzip on one
+# core would only make the window longer.
+set -euo pipefail
+
+cd {{ jar_lid_archive_dir }}
+tmp=$(mktemp -d {{ jar_lid_archive_dir }}/.dump.XXXXXX)
+trap 'rm -rf "$tmp"' EXIT
+
+docker compose exec -T db pg_dump -U {{ jar_lid_archive_postgres_user }} -d {{ jar_lid_archive_postgres_db }} -Fc > "$tmp/db.dump"
+tar -cf - -C "$tmp" db.dump -C {{ jar_lid_archive_dir }} seaweedfs

ansible/roles/jar_lid_archive/templates/env.j2

@@ -0,0 +1,5 @@
+# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
+# Read by docker compose to interpolate docker-compose.yml. 0600: secrets.
+POSTGRES_PASSWORD={{ jar_lid_archive_postgres_password }}
+S3_SECRET_KEY={{ jar_lid_archive_s3_secret_key }}
+DJANGO_SECRET_KEY={{ jar_lid_archive_django_secret_key }}