counterweight
c3a7ca0ba7
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
100 lines
3.4 KiB
YAML
100 lines
3.4 KiB
YAML
---
|
|
- name: Assert the deployment is fully specified
|
|
ansible.builtin.assert:
|
|
that:
|
|
- jar_lid_archive_domain | length > 0
|
|
- jar_lid_archive_django_secret_key | length >= 50
|
|
- jar_lid_archive_postgres_password | length > 0
|
|
- jar_lid_archive_s3_secret_key | length > 0
|
|
fail_msg: >-
|
|
jar_lid_archive needs a domain and its three secrets from the vault. The
|
|
app falls back to development defaults for every one of them, so an empty
|
|
value here would not fail - it would deploy with a published secret key.
|
|
no_log: true
|
|
|
|
- name: Create the deployment directory
|
|
ansible.builtin.file:
|
|
path: "{{ jar_lid_archive_dir }}"
|
|
state: directory
|
|
owner: root
|
|
group: root
|
|
mode: '0755'
|
|
|
|
# Existence only, no owner: the postgres and seaweedfs entrypoints chown their
|
|
# own data directories to the uid inside the container. Declaring an owner here
|
|
# would fight them on every run - the mempool MariaDB lesson.
|
|
- name: Ensure the data directories exist
|
|
ansible.builtin.file:
|
|
path: "{{ jar_lid_archive_dir }}/{{ item }}"
|
|
state: directory
|
|
loop:
|
|
- postgres
|
|
- seaweedfs
|
|
|
|
# diff: false, because --diff prints rendered content and this is all secrets.
|
|
- name: Write the compose environment file
|
|
ansible.builtin.template:
|
|
src: env.j2
|
|
dest: "{{ jar_lid_archive_dir }}/.env"
|
|
owner: root
|
|
group: root
|
|
mode: '0600'
|
|
diff: false
|
|
|
|
- name: Write docker-compose.yml
|
|
ansible.builtin.template:
|
|
src: docker-compose.yml.j2
|
|
dest: "{{ jar_lid_archive_dir }}/docker-compose.yml"
|
|
owner: root
|
|
group: root
|
|
mode: '0644'
|
|
|
|
- name: Log in to the container registry
|
|
ansible.builtin.command:
|
|
cmd: "docker login {{ jar_lid_archive_registry }} -u {{ jar_lid_archive_registry_user }} --password-stdin"
|
|
stdin: "{{ jar_lid_archive_registry_token }}"
|
|
when: jar_lid_archive_registry_token | length > 0
|
|
changed_when: false
|
|
no_log: true
|
|
|
|
# Pulled on every run: staging follows :master. `up` below recreates `web` only
|
|
# when the pulled digest differs from the running one.
|
|
- name: Pull images
|
|
ansible.builtin.command: docker compose pull --quiet
|
|
args:
|
|
chdir: "{{ jar_lid_archive_dir }}"
|
|
changed_when: false
|
|
|
|
# --wait blocks until every service's healthcheck passes, and `web`'s is
|
|
# /health/, which itself checks the database and the bucket - so this task
|
|
# succeeding means the app actually serves, not merely that containers exist.
|
|
- name: Start the stack and wait for it to be healthy
|
|
ansible.builtin.command: docker compose up -d --remove-orphans --wait --wait-timeout 300
|
|
args:
|
|
chdir: "{{ jar_lid_archive_dir }}"
|
|
register: jar_lid_archive_up
|
|
changed_when: jar_lid_archive_up.stderr is search('Created|Recreated|Started|Removed')
|
|
|
|
- name: Read the deployed version
|
|
ansible.builtin.uri:
|
|
url: "http://127.0.0.1:{{ jar_lid_archive_port }}/health/"
|
|
return_content: true
|
|
register: jar_lid_archive_health
|
|
# Under --check the stack was never started, so there is nothing to ask.
|
|
when: not ansible_check_mode
|
|
|
|
- name: Show the deployed version
|
|
ansible.builtin.debug:
|
|
msg: "{{ jar_lid_archive_health.json }}"
|
|
when: not ansible_check_mode
|
|
|
|
# Used by the backup (services/jar-lid-archive/setup_backup_jar_lid_archive.yml).
|
|
# Lives here because what to dump is a fact about this stack's layout.
|
|
- name: Install the dump script
|
|
ansible.builtin.template:
|
|
src: dump.sh.j2
|
|
dest: /usr/local/bin/jar-lid-archive-dump.sh
|
|
owner: root
|
|
group: root
|
|
mode: '0755'
|
|
validate: "bash -n %s"
|