personal_infra/ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml

52 lines
2.4 KiB
YAML
Raw Normal View History

jar-lid-archive: a staging deployment on its own box Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:56:30 +02:00
---
# Jar Lid Archive on its own box, published from the edge host.
#
# See roles/jar_lid_archive/README.md. Per-host values (domain, port, seed users,
# secrets) are in host_vars/<host>/, so a production host is a new inventory
# entry in [jar_lid_archive], not a new playbook.
- name: Deploy Jar Lid Archive
hosts: jar_lid_archive
become: yes
roles:
- jar_lid_archive
# "Does the app answer", as opposed to "are the containers up". /health/
# runs SELECT 1 and touches the bucket, and returns 503 if either fails.
- role: healthcheck
vars:
healthcheck_name: jar-lid-archive
healthcheck_description: "Jar Lid Archive /health/ (app, database and photo store)"
healthcheck_command: "curl -fsS --max-time 10 -o /dev/null http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_{{ jar_lid_archive_instance }}/external"
healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
- name: Publish Jar Lid Archive from the edge host
hosts: edge
become: yes
tasks:
# No basic auth: the one-off links (/links/...) are opened by people who have
# no account and no proxy credential. The app's own login guards the rest.
#
# Caddy sets X-Forwarded-Proto: https and overwrites any value the client
# sent, which is what lets the app trust it (DJANGO_BEHIND_TLS_PROXY).
#
# No request size limit is set, deliberately: Caddy has none by default, and
# phones upload multi-megabyte photos.
- name: Expose Jar Lid Archive over HTTPS
ansible.builtin.include_role:
name: caddy_site
vars:
caddy_site_name: "{{ hostvars[jla_host].jar_lid_archive_instance }}"
caddy_site_domain: "{{ hostvars[jla_host].jar_lid_archive_domain }}"
# Tailscale MagicDNS: the box has no public address.
caddy_site_upstream: "{{ hostvars[jla_host].ansible_host }}:{{ hostvars[jla_host].jar_lid_archive_port }}"
caddy_site_resolvers: "100.100.100.100"
loop: "{{ groups['jar_lid_archive'] }}"
# Not `item`: the role's own tasks loop, and would shadow it.
loop_control:
loop_var: jla_host
# The per-site validate cannot catch a conflict between sites.
- name: Validate the whole Caddyfile
ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
changed_when: false