jar-lid-archive: a staging deployment on its own box

Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ansible/roles/jar_lid_archive/README.md

@@ -0,0 +1,121 @@
+# `jar_lid_archive`
+
+Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive)
+as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and
+SeaweedFS as the S3 photo store (`storage`). The same three services as the app
+repo's own `compose.yaml`, so staging rehearses what dev and CI run.
+
+Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to
+every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local`
+(jar-lid-archive-staging-box, nodito VM 3300), public at
+`jars-staging.contrapeso.xyz` through vipy's Caddy.
+
+Per-host values are in `host_vars/<host>/main.yml`, and secrets are in `vault.yml`
+beside it. A production host would be a second inventory entry with its own
+`jar_lid_archive_instance`, domain and vault, not a second playbook.
+
+## Updates: follows `:master`
+
+CI pushes `:master` on every green build of master. Every run of the playbook
+runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only
+if the digest moved. There is no watchtower, so **a merge reaches staging when
+the playbook runs**, not on its own. The run ends by printing `/health/`, which
+includes the deployed version.
+
+`up --wait` only returns once every container's healthcheck passes. `web`'s is
+`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means
+the app actually serves.
+
+## Networking: why `web` is on the host network
+
+Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT`
+rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well
+as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids
+that, but at boot Docker races tailscaled for the address and loses.
+
+So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host
+port:
+
+- **LAN:** UFW default-deny blocks it.
+- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW
+  (checked on mempool-box), so the headscale ACL is the real control. It grants
+  `tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`.
+- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches
+  them from the host network.
+
+This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto`
+(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only
+thing that can reach gunicorn. Caddy overwrites whatever value the client sent.
+
+## Secrets and seed users
+
+The Django secret key, the Postgres password, the S3 secret and the three seed
+passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They
+reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose
+interpolates. `docker-compose.yml` itself has no secrets.
+
+The seed users are alice, bob and admin (superuser), with the fixture's usernames
+and **vault** passwords. The fixture's `*-test-pass` passwords are published in
+the app repo and this site is public. `seed_users` runs on every deploy and
+resets each listed user's password, so rotating one means editing the vault and
+re-running the playbook.
+
+```sh
+ansible-vault view host_vars/jar_lid_staging_local/vault.yml   # from ansible/
+```
+
+## Backups
+
+`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses
+`backup_source`. At 02:20 it stops `web` only, and
+`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with:
+
+- `db.dump`: `pg_dump -Fc`
+- `seaweedfs/`: the photo store's data directory, as files
+
+`web` is the only writer to the database and the photo store. With it stopped,
+both are quiet and are captured at the same instant. `storage` is deliberately
+left running: on this disk it can take minutes to restart, and the first test run
+that stopped it left the app down for 26 minutes. Uploads fail while `web` is
+down.
+
+The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally,
+and small-backups-box keeps three as well. Staging data is disposable.
+
+### Restore
+
+```sh
+age -d -i ~/.age/counterweight_age jar-lid-archive-staging_<ts>.tar.age | tar -xf - -C /tmp/restore
+# on the box, stack down:
+docker compose -f /opt/jar-lid-archive/docker-compose.yml down
+rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/
+docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db
+docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \
+  pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump
+docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait
+```
+
+## Monitoring
+
+| What | Where |
+|---|---|
+| Host liveness and disk | infra/400 (automatic for `managed`) |
+| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) |
+| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) |
+| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` |
+
+infra/401 has nothing to watch here, because no systemd units of ours run on
+this box.
+
+## Size and disk
+
+The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's
+random reads keep that mirror close to saturated, and this box has measured
+**~96% "full" I/O pressure** with nothing of its own running. Everything is slow
+here, but especially container starts. The healthcheck `start_period`s are sized
+for that. Moving the disk to `local-zfs` (NVMe) would fix it.
+
+
+1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian,
+the images, the photos and three days of local backups, and each backup is
+about the size of the whole photo store. infra/400's disk check alarms at 85%.

ansible/roles/jar_lid_archive/defaults/main.yml

@@ -0,0 +1,57 @@
+---
+# Jar Lid Archive: Django (gunicorn) + Postgres + SeaweedFS (S3) as one compose
+# stack. Source and CI live in forgejo.contrapeso.xyz/counterweight/jar-lid-archive.
+#
+# Per-deployment values (domain, port, retention) live in the host's host_vars,
+# because the Caddy play on the edge host and the backup plays read them too and
+# a role default cannot reach a second play.
+
+jar_lid_archive_dir: /opt/jar-lid-archive
+
+# ── Images ───────────────────────────────────────────────────────────────────
+# The app follows a moving tag on purpose: every run pulls it, and compose
+# recreates `web` only when the digest actually changed. CI pushes :master on
+# every green build of master, plus :<sha>.
+jar_lid_archive_image: forgejo.contrapeso.xyz/counterweight/jar-lid-archive
+jar_lid_archive_image_tag: master
+
+# Pinned to what the app's own compose.yaml and CI run, so staging rehearses the
+# same stack. Bump them together with the app repo, not separately.
+jar_lid_archive_postgres_image: "postgres:18"
+jar_lid_archive_seaweedfs_image: "chrislusf/seaweedfs:4.47"
+
+# Registry login. Empty = pull anonymously, which works today: the package is
+# owner-scoped in Forgejo and public even though the repository is private.
+jar_lid_archive_registry: forgejo.contrapeso.xyz
+jar_lid_archive_registry_user: ""
+jar_lid_archive_registry_token: ""
+
+# ── Networking ───────────────────────────────────────────────────────────────
+# gunicorn listens on the HOST network on this port. See the README for why host
+# networking rather than a published port: Docker's published ports bypass UFW.
+jar_lid_archive_port: 8000
+# Loopback-only ports for the two backing services, so the web container (on the
+# host network) can reach them and nothing else can.
+jar_lid_archive_postgres_port: 5432
+jar_lid_archive_s3_port: 8333
+
+jar_lid_archive_domain: ""              # e.g. jars-staging.contrapeso.xyz
+
+# ── App settings ─────────────────────────────────────────────────────────────
+jar_lid_archive_postgres_db: jarlid
+jar_lid_archive_postgres_user: jarlid
+jar_lid_archive_s3_access_key: jarlid
+jar_lid_archive_s3_bucket: photos
+
+# Secrets: pass from the vault. The assert in tasks/deploy.yml refuses empties.
+jar_lid_archive_django_secret_key: ""
+jar_lid_archive_postgres_password: ""
+jar_lid_archive_s3_secret_key: ""
+
+# ── Seed users ───────────────────────────────────────────────────────────────
+# Loaded with `manage.py seed_users -` on every run. That command RESETS each
+# listed user's password and flags, so the vault stays the source of truth.
+# Same usernames as the app's tests/fixtures/users.json, never its passwords:
+# this deployment is reachable from the internet.
+#   - {username: alice, password: "...", email: "...", is_superuser: false}
+jar_lid_archive_seed_users: []

ansible/roles/jar_lid_archive/tasks/deploy.yml

@@ -0,0 +1,100 @@
+---
+- name: Assert the deployment is fully specified
+  ansible.builtin.assert:
+    that:
+      - jar_lid_archive_domain | length > 0
+      - jar_lid_archive_django_secret_key | length >= 50
+      - jar_lid_archive_postgres_password | length > 0
+      - jar_lid_archive_s3_secret_key | length > 0
+    fail_msg: >-
+      jar_lid_archive needs a domain and its three secrets from the vault. The
+      app falls back to development defaults for every one of them, so an empty
+      value here would not fail - it would deploy with a published secret key.
+  no_log: true
+
+- name: Create the deployment directory
+  ansible.builtin.file:
+    path: "{{ jar_lid_archive_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: '0755'
+
+# Existence only, no owner: the postgres and seaweedfs entrypoints chown their
+# own data directories to the uid inside the container. Declaring an owner here
+# would fight them on every run - the mempool MariaDB lesson.
+- name: Ensure the data directories exist
+  ansible.builtin.file:
+    path: "{{ jar_lid_archive_dir }}/{{ item }}"
+    state: directory
+  loop:
+    - postgres
+    - seaweedfs
+
+# diff: false, because --diff prints rendered content and this is all secrets.
+- name: Write the compose environment file
+  ansible.builtin.template:
+    src: env.j2
+    dest: "{{ jar_lid_archive_dir }}/.env"
+    owner: root
+    group: root
+    mode: '0600'
+  diff: false
+
+- name: Write docker-compose.yml
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ jar_lid_archive_dir }}/docker-compose.yml"
+    owner: root
+    group: root
+    mode: '0644'
+
+- name: Log in to the container registry
+  ansible.builtin.command:
+    cmd: "docker login {{ jar_lid_archive_registry }} -u {{ jar_lid_archive_registry_user }} --password-stdin"
+    stdin: "{{ jar_lid_archive_registry_token }}"
+  when: jar_lid_archive_registry_token | length > 0
+  changed_when: false
+  no_log: true
+
+# Pulled on every run: staging follows :master. `up` below recreates `web` only
+# when the pulled digest differs from the running one.
+- name: Pull images
+  ansible.builtin.command: docker compose pull --quiet
+  args:
+    chdir: "{{ jar_lid_archive_dir }}"
+  changed_when: false
+
+# --wait blocks until every service's healthcheck passes, and `web`'s is
+# /health/, which itself checks the database and the bucket - so this task
+# succeeding means the app actually serves, not merely that containers exist.
+- name: Start the stack and wait for it to be healthy
+  ansible.builtin.command: docker compose up -d --remove-orphans --wait --wait-timeout 300
+  args:
+    chdir: "{{ jar_lid_archive_dir }}"
+  register: jar_lid_archive_up
+  changed_when: jar_lid_archive_up.stderr is search('Created|Recreated|Started|Removed')
+
+- name: Read the deployed version
+  ansible.builtin.uri:
+    url: "http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
+    return_content: true
+  register: jar_lid_archive_health
+  # Under --check the stack was never started, so there is nothing to ask.
+  when: not ansible_check_mode
+
+- name: Show the deployed version
+  ansible.builtin.debug:
+    msg: "{{ jar_lid_archive_health.json }}"
+  when: not ansible_check_mode
+
+# Used by the backup (services/jar-lid-archive/setup_backup_jar_lid_archive.yml).
+# Lives here because what to dump is a fact about this stack's layout.
+- name: Install the dump script
+  ansible.builtin.template:
+    src: dump.sh.j2
+    dest: /usr/local/bin/jar-lid-archive-dump.sh
+    owner: root
+    group: root
+    mode: '0755'
+    validate: "bash -n %s"

ansible/roles/jar_lid_archive/tasks/docker.yml

@@ -0,0 +1,74 @@
+---
+# Same Docker CE install as roles/mempool/tasks/docker.yml. infra/910_docker
+# targets every managed host, so each Docker role installs its own engine.
+- name: Remove old Docker-related packages
+  apt:
+    name:
+      - docker.io
+      - docker-doc
+      - docker-compose
+      - podman-docker
+      - containerd
+      - runc
+    state: absent
+    purge: yes
+    autoremove: yes
+
+- name: Update apt cache
+  apt:
+    update_cache: yes
+
+- name: Install prerequisites
+  apt:
+    name:
+      - ca-certificates
+      - curl
+    state: present
+
+- name: Create directory for Docker GPG key
+  file:
+    path: /etc/apt/keyrings
+    state: directory
+    mode: '0755'
+
+- name: Download Docker GPG key
+  get_url:
+    url: https://download.docker.com/linux/debian/gpg
+    dest: /etc/apt/keyrings/docker.asc
+    mode: '0644'
+
+- name: Get Debian architecture
+  command: dpkg --print-architecture
+  register: deb_arch
+  changed_when: false
+
+- name: Add Docker repository
+  apt_repository:
+    repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable"
+    filename: docker
+    state: present
+    update_cache: yes
+
+- name: Install Docker packages
+  apt:
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-buildx-plugin
+      - docker-compose-plugin
+    state: present
+    update_cache: yes
+
+- name: Ensure Docker is started and enabled
+  systemd:
+    name: docker
+    enabled: yes
+    state: started
+
+- name: Add user to docker group
+  user:
+    name: "{{ ansible_user }}"
+    groups: docker
+    append: yes
+

ansible/roles/jar_lid_archive/tasks/main.yml

@@ -0,0 +1,6 @@
+---
+# import_tasks, not include_tasks: static imports stay visible to --list-tasks,
+# which is the main verification tool in this repo.
+- ansible.builtin.import_tasks: docker.yml
+- ansible.builtin.import_tasks: deploy.yml
+- ansible.builtin.import_tasks: seed.yml

ansible/roles/jar_lid_archive/tasks/seed.yml

@@ -0,0 +1,13 @@
+---
+# seed_users is idempotent but RESETS each listed user's password and flags, so
+# re-running it is how a rotated vault password reaches the app. It reports
+# "N created, M updated"; only a creation counts as a change.
+- name: Seed users
+  ansible.builtin.command:
+    cmd: docker compose exec -T web python manage.py seed_users -
+    chdir: "{{ jar_lid_archive_dir }}"
+    stdin: "{{ jar_lid_archive_seed_users | to_json }}"
+  register: jar_lid_archive_seed
+  changed_when: "jar_lid_archive_seed.stdout is not search('Seeded users: 0 created')"
+  when: jar_lid_archive_seed_users | length > 0
+  no_log: true

ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2

@@ -0,0 +1,123 @@
+# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
+#
+# Same three services as the app repo's compose.yaml, with three differences:
+#   - `web` runs the CI-built image instead of `build: .`
+#   - `web` is on the HOST network, so UFW and the tailnet ACL decide who reaches
+#     gunicorn. A published port would be DNATed by Docker straight past UFW and
+#     answer on the LAN too. See the role README.
+#   - data is bind-mounted under {{ jar_lid_archive_dir }} rather than in named
+#     volumes, so the backup tars a path it can name.
+#
+# Secrets are interpolated from .env (0600) beside this file; this file has none.
+name: jar-lid-archive
+
+x-logging: &logging
+  # Docker's default json-file driver never rotates, on a 10 GB disk.
+  driver: json-file
+  options:
+    max-size: "10m"
+    max-file: "3"
+
+x-postgres-env: &postgres-env
+  POSTGRES_DB: "{{ jar_lid_archive_postgres_db }}"
+  POSTGRES_USER: "{{ jar_lid_archive_postgres_user }}"
+  POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?}
+
+x-s3-env: &s3-env
+  S3_ACCESS_KEY: "{{ jar_lid_archive_s3_access_key }}"
+  S3_SECRET_KEY: ${S3_SECRET_KEY:?}
+
+services:
+  web:
+    image: "{{ jar_lid_archive_image }}:{{ jar_lid_archive_image_tag }}"
+    # gunicorn binds 0.0.0.0:{{ jar_lid_archive_port }} - the port is fixed by the image's CMD.
+    network_mode: host
+    environment:
+      <<: [*postgres-env, *s3-env]
+      POSTGRES_HOST: 127.0.0.1
+      POSTGRES_PORT: "{{ jar_lid_archive_postgres_port }}"
+      S3_ENDPOINT_URL: "http://127.0.0.1:{{ jar_lid_archive_s3_port }}"
+      S3_BUCKET: "{{ jar_lid_archive_s3_bucket }}"
+      DJANGO_DEBUG: "0"
+      DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY:?}
+      # 127.0.0.1 for the container healthcheck and the probe, which both call
+      # /health/ directly rather than through Caddy.
+      DJANGO_ALLOWED_HOSTS: "{{ jar_lid_archive_domain }},127.0.0.1,localhost"
+      # Caddy terminates TLS and forwards plain HTTP with X-Forwarded-Proto. The
+      # app must trust that header or every POST fails CSRF and share links come
+      # out http://. Safe only because nothing but Caddy can reach gunicorn.
+      DJANGO_BEHIND_TLS_PROXY: "1"
+      DJANGO_CSRF_TRUSTED_ORIGINS: "https://{{ jar_lid_archive_domain }}"
+    depends_on:
+      db:
+        condition: service_healthy
+      storage:
+        condition: service_healthy
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:{{ jar_lid_archive_port }}/health/')"]
+      interval: 30s
+      timeout: 5s
+      retries: 5
+      start_period: 180s
+    logging: *logging
+    restart: unless-stopped
+
+  db:
+    image: "{{ jar_lid_archive_postgres_image }}"
+    environment: *postgres-env
+    ports:
+      - "127.0.0.1:{{ jar_lid_archive_postgres_port }}:5432"
+    volumes:
+      - ./postgres:/var/lib/postgresql
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
+      interval: 5s
+      timeout: 3s
+      retries: 15
+      start_period: 120s
+    logging: *logging
+    restart: unless-stopped
+
+  storage:
+    image: "{{ jar_lid_archive_seaweedfs_image }}"
+    environment: *s3-env
+    # Copied from the app's compose.yaml: write the S3 identity from the
+    # environment, then hand over to the image's entrypoint, which drops root
+    # and runs master + volume + filer + S3 gateway.
+    #
+    # The `rm -f` is ours, and load-bearing. On a RESTART of the same container
+    # (a reboot, a backup's stop/start) /tmp/s3.json survives, owned by seaweed
+    # from the chown below. Root cannot then re-create it: /tmp is sticky and
+    # the kernel's protected_regular refuses O_CREAT on another user's file
+    # there. The result was "can't create /tmp/s3.json: Permission denied", an
+    # unhealthy storage container, and web never starting - found on the first
+    # backup run.
+    entrypoint: ["sh", "-c"]
+    command:
+      - |
+        rm -f /tmp/s3.json
+        umask 077
+        cat > /tmp/s3.json <<EOF
+        {"identities": [{"name": "app",
+          "credentials": [{"accessKey": "$$S3_ACCESS_KEY", "secretKey": "$$S3_SECRET_KEY"}],
+          "actions": ["Admin", "Read", "Write", "List", "Tagging"]}]}
+        EOF
+        chown seaweed /tmp/s3.json
+        exec /entrypoint.sh server -s3 -s3.config=/tmp/s3.json
+    ports:
+      - "127.0.0.1:{{ jar_lid_archive_s3_port }}:8333"
+    volumes:
+      - ./seaweedfs:/data
+    # start_period, because this VM's disk is on nodito's HDD mirror, shared
+    # with knots-box. Under that contention SeaweedFS has taken over the 150s
+    # that retries x interval allowed; compose then marked it unhealthy, and
+    # web's depends_on gave up and never started it - a 26-minute outage.
+    # Failures inside start_period do not count.
+    healthcheck:
+      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8333/healthz"]
+      interval: 5s
+      timeout: 3s
+      retries: 30
+      start_period: 300s
+    logging: *logging
+    restart: unless-stopped

ansible/roles/jar_lid_archive/templates/dump.sh.j2

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Jar Lid Archive dump - managed by Ansible (roles/jar_lid_archive).
+#
+# Writes ONE uncompressed tar to STDOUT, for roles/backup_source to encrypt:
+#   db.dump      pg_dump custom format   -> pg_restore -d jarlid db.dump
+#   seaweedfs/   the photo store, as files
+#
+# Called with `web` stopped (the backup playbook arranges it). web is the only
+# writer to db and storage, so both are quiescent and agree with each other: no
+# row points at a photo the tar does not have. storage keeps running; with no
+# writes arriving its volume files do not change under tar.
+#
+# The pg_dump lands in a temp dir on the same disk as the live database, which
+# is already plaintext; it is gone before this script exits.
+#
+# tar, not tar.gz: the bulk is JPEGs, which do not compress, and gzip on one
+# core would only make the window longer.
+set -euo pipefail
+
+cd {{ jar_lid_archive_dir }}
+tmp=$(mktemp -d {{ jar_lid_archive_dir }}/.dump.XXXXXX)
+trap 'rm -rf "$tmp"' EXIT
+
+docker compose exec -T db pg_dump -U {{ jar_lid_archive_postgres_user }} -d {{ jar_lid_archive_postgres_db }} -Fc > "$tmp/db.dump"
+tar -cf - -C "$tmp" db.dump -C {{ jar_lid_archive_dir }} seaweedfs

ansible/roles/jar_lid_archive/templates/env.j2

@@ -0,0 +1,5 @@
+# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
+# Read by docker compose to interpolate docker-compose.yml. 0600: secrets.
+POSTGRES_PASSWORD={{ jar_lid_archive_postgres_password }}
+S3_SECRET_KEY={{ jar_lid_archive_s3_secret_key }}
+DJANGO_SECRET_KEY={{ jar_lid_archive_django_secret_key }}