personal_infra/ansible/roles/jar_lid_archive/README.md

# `jar_lid_archive`

Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive)
as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and
SeaweedFS as the S3 photo store (`storage`). The same three services as the app
repo's own `compose.yaml`, so staging rehearses what dev and CI run.

Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to
every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local`
(jar-lid-archive-staging-box, nodito VM 3300), public at
`jars-staging.contrapeso.xyz` through vipy's Caddy.

Per-host values are in `host_vars/<host>/main.yml`, and secrets are in `vault.yml`
beside it. A production host would be a second inventory entry with its own
`jar_lid_archive_instance`, domain and vault, not a second playbook.

## Updates: follows `:master`

CI pushes `:master` on every green build of master. Every run of the playbook
runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only
if the digest moved. There is no watchtower, so **a merge reaches staging when
the playbook runs**, not on its own. The run ends by printing `/health/`, which
includes the deployed version.

`up --wait` only returns once every container's healthcheck passes. `web`'s is
`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means
the app actually serves.

## Networking: why `web` is on the host network

Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT`
rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well
as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids
that, but at boot Docker races tailscaled for the address and loses.

So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host
port:

- **LAN:** UFW default-deny blocks it.
- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW
  (checked on mempool-box), so the headscale ACL is the real control. It grants
  `tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`.
- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches
  them from the host network.

This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto`
(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only
thing that can reach gunicorn. Caddy overwrites whatever value the client sent.

## Secrets and seed users

The Django secret key, the Postgres password, the S3 secret and the three seed
passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They
reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose
interpolates. `docker-compose.yml` itself has no secrets.

The seed users are alice, bob and admin (superuser), with the fixture's usernames
and **vault** passwords. The fixture's `*-test-pass` passwords are published in
the app repo and this site is public. `seed_users` runs on every deploy and
resets each listed user's password, so rotating one means editing the vault and
re-running the playbook.

```sh
ansible-vault view host_vars/jar_lid_staging_local/vault.yml   # from ansible/
```

## Backups

`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses
`backup_source`. At 02:20 it stops `web` only, and
`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with:

- `db.dump`: `pg_dump -Fc`
- `seaweedfs/`: the photo store's data directory, as files

`web` is the only writer to the database and the photo store. With it stopped,
both are quiet and are captured at the same instant. `storage` is deliberately
left running: on this disk it can take minutes to restart, and the first test run
that stopped it left the app down for 26 minutes. Uploads fail while `web` is
down.

The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally,
and small-backups-box keeps three as well. Staging data is disposable.

### Restore

```sh
age -d -i ~/.age/counterweight_age jar-lid-archive-staging_<ts>.tar.age | tar -xf - -C /tmp/restore
# on the box, stack down:
docker compose -f /opt/jar-lid-archive/docker-compose.yml down
rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db
docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \
  pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump
docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait
```

## Monitoring

| What | Where |
|---|---|
| Host liveness and disk | infra/400 (automatic for `managed`) |
| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) |
| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) |
| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` |

infra/401 has nothing to watch here, because no systemd units of ours run on
this box.

## Size and disk

The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's
random reads keep that mirror close to saturated, and this box has measured
**~96% "full" I/O pressure** with nothing of its own running. Everything is slow
here, but especially container starts. The healthcheck `start_period`s are sized
for that. Moving the disk to `local-zfs` (NVMe) would fix it.


1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian,
the images, the photos and three days of local backups, and each backup is
about the size of the whole photo store. infra/400's disk check alarms at 85%.