counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
personal_infra/ansible/roles/jar_lid_archive/README.md
counterweight c3a7ca0ba7
jar-lid-archive: a staging deployment on its own box 
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:56:30 +02:00

5.5 KiB
Raw Blame History

jar_lid_archive

Runs Jar Lid Archive as a docker compose stack: Django under gunicorn (web), Postgres 18 (db) and SeaweedFS as the S3 photo store (storage). The same three services as the app repo's own compose.yaml, so staging rehearses what dev and CI run.

Deployed by services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml to every host in [jar_lid_archive]. Today that is jar_lid_staging_local (jar-lid-archive-staging-box, nodito VM 3300), public at jars-staging.contrapeso.xyz through vipy's Caddy.

Per-host values are in host_vars/<host>/main.yml, and secrets are in vault.yml beside it. A production host would be a second inventory entry with its own jar_lid_archive_instance, domain and vault, not a second playbook.

Updates: follows :master

CI pushes :master on every green build of master. Every run of the playbook runs docker compose pull and up -d --wait, and compose recreates web only if the digest moved. There is no watchtower, so a merge reaches staging when the playbook runs, not on its own. The run ends by printing /health/, which includes the deployed version.

up --wait only returns once every container's healthcheck passes. web's is /health/, which runs SELECT 1 and touches the bucket, so a green run means the app actually serves.

Networking: why web is on the host network

Docker's published ports are DNATed in the nat table, before UFW's INPUT rules ever see the packet. A ports: "8000:8000" would answer on the LAN as well as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids that, but at boot Docker races tailscaled for the address and loses.

So web uses network_mode: host, and gunicorn's port is an ordinary host port:

  • LAN: UFW default-deny blocks it.
  • Tailnet: tailscaled's ts-input chain accepts tailscale0 ahead of UFW (checked on mempool-box), so the headscale ACL is the real control. It grants tag:prd-vipy exactly tag:jar-lid-archive-staging-box:8000.
  • db and storage publish on 127.0.0.1 only, which is how web reaches them from the host network.

This is load-bearing for HTTPS. The app is told to trust X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the only thing that can reach gunicorn. Caddy overwrites whatever value the client sent.

Secrets and seed users

The Django secret key, the Postgres password, the S3 secret and the three seed passwords were generated into host_vars/jar_lid_staging_local/vault.yml. They reach the box as /opt/jar-lid-archive/.env (0600, root), which compose interpolates. docker-compose.yml itself has no secrets.

The seed users are alice, bob and admin (superuser), with the fixture's usernames and vault passwords. The fixture's *-test-pass passwords are published in the app repo and this site is public. seed_users runs on every deploy and resets each listed user's password, so rotating one means editing the vault and re-running the playbook.

ansible-vault view host_vars/jar_lid_staging_local/vault.yml   # from ansible/

Backups

services/jar-lid-archive/setup_backup_jar_lid_archive.yml uses backup_source. At 02:20 it stops web only, and /usr/local/bin/jar-lid-archive-dump.sh writes one tar with:

  • db.dump: pg_dump -Fc
  • seaweedfs/: the photo store's data directory, as files

web is the only writer to the database and the photo store. With it stopped, both are quiet and are captured at the same instant. storage is deliberately left running: on this disk it can take minutes to restart, and the first test run that stopped it left the app down for 26 minutes. Uploads fail while web is down.

The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally, and small-backups-box keeps three as well. Staging data is disposable.

Restore

age -d -i ~/.age/counterweight_age jar-lid-archive-staging_<ts>.tar.age | tar -xf - -C /tmp/restore
# on the box, stack down:
docker compose -f /opt/jar-lid-archive/docker-compose.yml down
rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db
docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \
  pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait

Monitoring

What Where
Host liveness and disk infra/400 (automatic for managed)
/health/ from the box, every 5 min probe_jar-lid-archive-staging (403)
/health/ through DNS, Caddy, TLS and the tailnet jars-staging public site (402)
Dump ran / artefact arrived backup-dump_ / backup-store_jar-lid-archive-staging

infra/401 has nothing to watch here, because no systemd units of ours run on this box.

Size and disk

The VM's disk is on proxmox-tank-1, nodito's mirror of two HDDs. knots-box's random reads keep that mirror close to saturated, and this box has measured ~96% "full" I/O pressure with nothing of its own running. Everything is slow here, but especially container starts. The healthcheck start_periods are sized for that. Moving the disk to local-zfs (NVMe) would fix it.

1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian, the images, the photos and three days of local backups, and each backup is about the size of the whole photo store. infra/400's disk check alarms at 85%.