counterweight
c3a7ca0ba7
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
121 lines
5.5 KiB
Markdown
121 lines
5.5 KiB
Markdown
# `jar_lid_archive`
|
|
|
|
Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive)
|
|
as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and
|
|
SeaweedFS as the S3 photo store (`storage`). The same three services as the app
|
|
repo's own `compose.yaml`, so staging rehearses what dev and CI run.
|
|
|
|
Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to
|
|
every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local`
|
|
(jar-lid-archive-staging-box, nodito VM 3300), public at
|
|
`jars-staging.contrapeso.xyz` through vipy's Caddy.
|
|
|
|
Per-host values are in `host_vars/<host>/main.yml`, and secrets are in `vault.yml`
|
|
beside it. A production host would be a second inventory entry with its own
|
|
`jar_lid_archive_instance`, domain and vault, not a second playbook.
|
|
|
|
## Updates: follows `:master`
|
|
|
|
CI pushes `:master` on every green build of master. Every run of the playbook
|
|
runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only
|
|
if the digest moved. There is no watchtower, so **a merge reaches staging when
|
|
the playbook runs**, not on its own. The run ends by printing `/health/`, which
|
|
includes the deployed version.
|
|
|
|
`up --wait` only returns once every container's healthcheck passes. `web`'s is
|
|
`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means
|
|
the app actually serves.
|
|
|
|
## Networking: why `web` is on the host network
|
|
|
|
Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT`
|
|
rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well
|
|
as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids
|
|
that, but at boot Docker races tailscaled for the address and loses.
|
|
|
|
So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host
|
|
port:
|
|
|
|
- **LAN:** UFW default-deny blocks it.
|
|
- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW
|
|
(checked on mempool-box), so the headscale ACL is the real control. It grants
|
|
`tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`.
|
|
- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches
|
|
them from the host network.
|
|
|
|
This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto`
|
|
(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only
|
|
thing that can reach gunicorn. Caddy overwrites whatever value the client sent.
|
|
|
|
## Secrets and seed users
|
|
|
|
The Django secret key, the Postgres password, the S3 secret and the three seed
|
|
passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They
|
|
reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose
|
|
interpolates. `docker-compose.yml` itself has no secrets.
|
|
|
|
The seed users are alice, bob and admin (superuser), with the fixture's usernames
|
|
and **vault** passwords. The fixture's `*-test-pass` passwords are published in
|
|
the app repo and this site is public. `seed_users` runs on every deploy and
|
|
resets each listed user's password, so rotating one means editing the vault and
|
|
re-running the playbook.
|
|
|
|
```sh
|
|
ansible-vault view host_vars/jar_lid_staging_local/vault.yml # from ansible/
|
|
```
|
|
|
|
## Backups
|
|
|
|
`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses
|
|
`backup_source`. At 02:20 it stops `web` only, and
|
|
`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with:
|
|
|
|
- `db.dump`: `pg_dump -Fc`
|
|
- `seaweedfs/`: the photo store's data directory, as files
|
|
|
|
`web` is the only writer to the database and the photo store. With it stopped,
|
|
both are quiet and are captured at the same instant. `storage` is deliberately
|
|
left running: on this disk it can take minutes to restart, and the first test run
|
|
that stopped it left the app down for 26 minutes. Uploads fail while `web` is
|
|
down.
|
|
|
|
The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally,
|
|
and small-backups-box keeps three as well. Staging data is disposable.
|
|
|
|
### Restore
|
|
|
|
```sh
|
|
age -d -i ~/.age/counterweight_age jar-lid-archive-staging_<ts>.tar.age | tar -xf - -C /tmp/restore
|
|
# on the box, stack down:
|
|
docker compose -f /opt/jar-lid-archive/docker-compose.yml down
|
|
rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/
|
|
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db
|
|
docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \
|
|
pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump
|
|
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait
|
|
```
|
|
|
|
## Monitoring
|
|
|
|
| What | Where |
|
|
|---|---|
|
|
| Host liveness and disk | infra/400 (automatic for `managed`) |
|
|
| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) |
|
|
| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) |
|
|
| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` |
|
|
|
|
infra/401 has nothing to watch here, because no systemd units of ours run on
|
|
this box.
|
|
|
|
## Size and disk
|
|
|
|
The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's
|
|
random reads keep that mirror close to saturated, and this box has measured
|
|
**~96% "full" I/O pressure** with nothing of its own running. Everything is slow
|
|
here, but especially container starts. The healthcheck `start_period`s are sized
|
|
for that. Moving the disk to `local-zfs` (NVMe) would fix it.
|
|
|
|
|
|
1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian,
|
|
the images, the photos and three days of local backups, and each backup is
|
|
about the size of the whole photo store. infra/400's disk check alarms at 85%.
|