jar-lid-archive: a staging deployment on its own box
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
97c7d580bc
commit
c3a7ca0ba7
GPG key ID:
883EDBAA726BD96C
22 changed files with 929 additions and 215 deletions
123
ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2
Normal file
123
ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host.
|
||||
#
|
||||
# Same three services as the app repo's compose.yaml, with three differences:
|
||||
# - `web` runs the CI-built image instead of `build: .`
|
||||
# - `web` is on the HOST network, so UFW and the tailnet ACL decide who reaches
|
||||
# gunicorn. A published port would be DNATed by Docker straight past UFW and
|
||||
# answer on the LAN too. See the role README.
|
||||
# - data is bind-mounted under {{ jar_lid_archive_dir }} rather than in named
|
||||
# volumes, so the backup tars a path it can name.
|
||||
#
|
||||
# Secrets are interpolated from .env (0600) beside this file; this file has none.
|
||||
name: jar-lid-archive
|
||||
|
||||
x-logging: &logging
|
||||
# Docker's default json-file driver never rotates, on a 10 GB disk.
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
|
||||
x-postgres-env: &postgres-env
|
||||
POSTGRES_DB: "{{ jar_lid_archive_postgres_db }}"
|
||||
POSTGRES_USER: "{{ jar_lid_archive_postgres_user }}"
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?}
|
||||
|
||||
x-s3-env: &s3-env
|
||||
S3_ACCESS_KEY: "{{ jar_lid_archive_s3_access_key }}"
|
||||
S3_SECRET_KEY: ${S3_SECRET_KEY:?}
|
||||
|
||||
services:
|
||||
web:
|
||||
image: "{{ jar_lid_archive_image }}:{{ jar_lid_archive_image_tag }}"
|
||||
# gunicorn binds 0.0.0.0:{{ jar_lid_archive_port }} - the port is fixed by the image's CMD.
|
||||
network_mode: host
|
||||
environment:
|
||||
<<: [*postgres-env, *s3-env]
|
||||
POSTGRES_HOST: 127.0.0.1
|
||||
POSTGRES_PORT: "{{ jar_lid_archive_postgres_port }}"
|
||||
S3_ENDPOINT_URL: "http://127.0.0.1:{{ jar_lid_archive_s3_port }}"
|
||||
S3_BUCKET: "{{ jar_lid_archive_s3_bucket }}"
|
||||
DJANGO_DEBUG: "0"
|
||||
DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY:?}
|
||||
# 127.0.0.1 for the container healthcheck and the probe, which both call
|
||||
# /health/ directly rather than through Caddy.
|
||||
DJANGO_ALLOWED_HOSTS: "{{ jar_lid_archive_domain }},127.0.0.1,localhost"
|
||||
# Caddy terminates TLS and forwards plain HTTP with X-Forwarded-Proto. The
|
||||
# app must trust that header or every POST fails CSRF and share links come
|
||||
# out http://. Safe only because nothing but Caddy can reach gunicorn.
|
||||
DJANGO_BEHIND_TLS_PROXY: "1"
|
||||
DJANGO_CSRF_TRUSTED_ORIGINS: "https://{{ jar_lid_archive_domain }}"
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
storage:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:{{ jar_lid_archive_port }}/health/')"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 180s
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
db:
|
||||
image: "{{ jar_lid_archive_postgres_image }}"
|
||||
environment: *postgres-env
|
||||
ports:
|
||||
- "127.0.0.1:{{ jar_lid_archive_postgres_port }}:5432"
|
||||
volumes:
|
||||
- ./postgres:/var/lib/postgresql
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 15
|
||||
start_period: 120s
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
|
||||
storage:
|
||||
image: "{{ jar_lid_archive_seaweedfs_image }}"
|
||||
environment: *s3-env
|
||||
# Copied from the app's compose.yaml: write the S3 identity from the
|
||||
# environment, then hand over to the image's entrypoint, which drops root
|
||||
# and runs master + volume + filer + S3 gateway.
|
||||
#
|
||||
# The `rm -f` is ours, and load-bearing. On a RESTART of the same container
|
||||
# (a reboot, a backup's stop/start) /tmp/s3.json survives, owned by seaweed
|
||||
# from the chown below. Root cannot then re-create it: /tmp is sticky and
|
||||
# the kernel's protected_regular refuses O_CREAT on another user's file
|
||||
# there. The result was "can't create /tmp/s3.json: Permission denied", an
|
||||
# unhealthy storage container, and web never starting - found on the first
|
||||
# backup run.
|
||||
entrypoint: ["sh", "-c"]
|
||||
command:
|
||||
- |
|
||||
rm -f /tmp/s3.json
|
||||
umask 077
|
||||
cat > /tmp/s3.json <<EOF
|
||||
{"identities": [{"name": "app",
|
||||
"credentials": [{"accessKey": "$$S3_ACCESS_KEY", "secretKey": "$$S3_SECRET_KEY"}],
|
||||
"actions": ["Admin", "Read", "Write", "List", "Tagging"]}]}
|
||||
EOF
|
||||
chown seaweed /tmp/s3.json
|
||||
exec /entrypoint.sh server -s3 -s3.config=/tmp/s3.json
|
||||
ports:
|
||||
- "127.0.0.1:{{ jar_lid_archive_s3_port }}:8333"
|
||||
volumes:
|
||||
- ./seaweedfs:/data
|
||||
# start_period, because this VM's disk is on nodito's HDD mirror, shared
|
||||
# with knots-box. Under that contention SeaweedFS has taken over the 150s
|
||||
# that retries x interval allowed; compose then marked it unhealthy, and
|
||||
# web's depends_on gave up and never started it - a 26-minute outage.
|
||||
# Failures inside start_period do not count.
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8333/healthz"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
start_period: 300s
|
||||
logging: *logging
|
||||
restart: unless-stopped
|
||||
Loading…
Add table
Add a link
Reference in a new issue