jar-lid-archive: a staging deployment on its own box
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
97c7d580bc
commit
c3a7ca0ba7
GPG key ID:
883EDBAA726BD96C
22 changed files with 929 additions and 215 deletions
121
ansible/roles/jar_lid_archive/README.md
Normal file
121
ansible/roles/jar_lid_archive/README.md
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
# `jar_lid_archive`
|
||||
|
||||
Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive)
|
||||
as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and
|
||||
SeaweedFS as the S3 photo store (`storage`). The same three services as the app
|
||||
repo's own `compose.yaml`, so staging rehearses what dev and CI run.
|
||||
|
||||
Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to
|
||||
every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local`
|
||||
(jar-lid-archive-staging-box, nodito VM 3300), public at
|
||||
`jars-staging.contrapeso.xyz` through vipy's Caddy.
|
||||
|
||||
Per-host values are in `host_vars/<host>/main.yml`, and secrets are in `vault.yml`
|
||||
beside it. A production host would be a second inventory entry with its own
|
||||
`jar_lid_archive_instance`, domain and vault, not a second playbook.
|
||||
|
||||
## Updates: follows `:master`
|
||||
|
||||
CI pushes `:master` on every green build of master. Every run of the playbook
|
||||
runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only
|
||||
if the digest moved. There is no watchtower, so **a merge reaches staging when
|
||||
the playbook runs**, not on its own. The run ends by printing `/health/`, which
|
||||
includes the deployed version.
|
||||
|
||||
`up --wait` only returns once every container's healthcheck passes. `web`'s is
|
||||
`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means
|
||||
the app actually serves.
|
||||
|
||||
## Networking: why `web` is on the host network
|
||||
|
||||
Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT`
|
||||
rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well
|
||||
as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids
|
||||
that, but at boot Docker races tailscaled for the address and loses.
|
||||
|
||||
So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host
|
||||
port:
|
||||
|
||||
- **LAN:** UFW default-deny blocks it.
|
||||
- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW
|
||||
(checked on mempool-box), so the headscale ACL is the real control. It grants
|
||||
`tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`.
|
||||
- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches
|
||||
them from the host network.
|
||||
|
||||
This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto`
|
||||
(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only
|
||||
thing that can reach gunicorn. Caddy overwrites whatever value the client sent.
|
||||
|
||||
## Secrets and seed users
|
||||
|
||||
The Django secret key, the Postgres password, the S3 secret and the three seed
|
||||
passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They
|
||||
reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose
|
||||
interpolates. `docker-compose.yml` itself has no secrets.
|
||||
|
||||
The seed users are alice, bob and admin (superuser), with the fixture's usernames
|
||||
and **vault** passwords. The fixture's `*-test-pass` passwords are published in
|
||||
the app repo and this site is public. `seed_users` runs on every deploy and
|
||||
resets each listed user's password, so rotating one means editing the vault and
|
||||
re-running the playbook.
|
||||
|
||||
```sh
|
||||
ansible-vault view host_vars/jar_lid_staging_local/vault.yml # from ansible/
|
||||
```
|
||||
|
||||
## Backups
|
||||
|
||||
`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses
|
||||
`backup_source`. At 02:20 it stops `web` only, and
|
||||
`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with:
|
||||
|
||||
- `db.dump`: `pg_dump -Fc`
|
||||
- `seaweedfs/`: the photo store's data directory, as files
|
||||
|
||||
`web` is the only writer to the database and the photo store. With it stopped,
|
||||
both are quiet and are captured at the same instant. `storage` is deliberately
|
||||
left running: on this disk it can take minutes to restart, and the first test run
|
||||
that stopped it left the app down for 26 minutes. Uploads fail while `web` is
|
||||
down.
|
||||
|
||||
The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally,
|
||||
and small-backups-box keeps three as well. Staging data is disposable.
|
||||
|
||||
### Restore
|
||||
|
||||
```sh
|
||||
age -d -i ~/.age/counterweight_age jar-lid-archive-staging_<ts>.tar.age | tar -xf - -C /tmp/restore
|
||||
# on the box, stack down:
|
||||
docker compose -f /opt/jar-lid-archive/docker-compose.yml down
|
||||
rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/
|
||||
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db
|
||||
docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \
|
||||
pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump
|
||||
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
| What | Where |
|
||||
|---|---|
|
||||
| Host liveness and disk | infra/400 (automatic for `managed`) |
|
||||
| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) |
|
||||
| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) |
|
||||
| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` |
|
||||
|
||||
infra/401 has nothing to watch here, because no systemd units of ours run on
|
||||
this box.
|
||||
|
||||
## Size and disk
|
||||
|
||||
The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's
|
||||
random reads keep that mirror close to saturated, and this box has measured
|
||||
**~96% "full" I/O pressure** with nothing of its own running. Everything is slow
|
||||
here, but especially container starts. The healthcheck `start_period`s are sized
|
||||
for that. Moving the disk to `local-zfs` (NVMe) would fix it.
|
||||
|
||||
|
||||
1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian,
|
||||
the images, the photos and three days of local backups, and each backup is
|
||||
about the size of the whole photo store. infra/400's disk check alarms at 85%.
|
||||
Loading…
Add table
Add a link
Reference in a new issue