counterweight
c3a7ca0ba7
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
85 lines
3.9 KiB
YAML
85 lines
3.9 KiB
YAML
new_user: counterweight
|
|
ssh_port: 22
|
|
allow_ssh_from: "any"
|
|
root_domain: contrapeso.xyz
|
|
|
|
# Uptime Kuma was decommissioned on 2026-09-11. The monitoring blocks in the
|
|
# playbooks are kept deliberately — the check logic is meant to be rewired to
|
|
# whatever replaces it. This flag keeps them inert until then. See archive/uptime_kuma/.
|
|
|
|
# age recipient for all backup artefacts
|
|
age_backup_recipient: "age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza"
|
|
|
|
# Public key small-backups-box pulls with
|
|
# Authorised on each source host for an unprivileged, dedicated user only
|
|
backup_pull_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfIixKMhA9z+Nvyx6ToZIniC8aEgyiInRiboaTTemgX offsite-backup-pull"
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Subdomains. Global because the edge host proxies for services that live on
|
|
# other machines, so no single inventory group covers the readers. Combine with
|
|
# root_domain above to build an FQDN.
|
|
#
|
|
# Moved here from services_config.yml, which 30 plays had to remember to name in
|
|
# vars_files: - a file everyone must opt into is a file someone will forget.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
subdomains:
|
|
# Monitoring
|
|
gatus: status
|
|
|
|
# VPN infrastructure (spacey)
|
|
headscale: headscale
|
|
|
|
# Core services (vipy)
|
|
vaultwarden: vault
|
|
forgejo: forgejo
|
|
lnbits: wallet
|
|
|
|
# Secondary services (vipy)
|
|
ntfy_emergency_app: avisame
|
|
personal_blog: pablohere
|
|
|
|
# Memos (memos-box)
|
|
memos: memos
|
|
|
|
# Mempool block explorer (mempool-box, proxied via vipy)
|
|
mempool: mempool
|
|
|
|
# DATUM Gateway dashboard (knots-box, proxied via vipy)
|
|
datum_gateway: datum
|
|
|
|
# Monero restricted RPC (monerod-box, proxied via vipy). Not a dashboard:
|
|
# this is the JSON-RPC endpoint Feather points at. Caddy terminates TLS and
|
|
# monerod's own Digest auth does the authentication.
|
|
monero: monero
|
|
|
|
# Monero block explorer (monerod-box, proxied via vipy). Fully public and
|
|
# unauthenticated, like any block explorer - the data is the public chain.
|
|
monero_explorer: exmrplore
|
|
|
|
# Navidrome music server (music-box, proxied via vipy). No basic auth:
|
|
# Navidrome has its own login, and the Subsonic API the mobile clients speak
|
|
# has nowhere to carry a second credential. Same call made for monero above.
|
|
music: music
|
|
|
|
# Jar Lid Archive staging (jar-lid-archive-staging-box, proxied via vipy).
|
|
# Public with no basic auth: the one-off photo links are for people without
|
|
# an account, and the app has its own login for everything else.
|
|
jar_lid_archive_staging: jars-staging
|
|
|
|
# Read by plays across several groups, so global rather than group_vars/<group>.
|
|
headscale_namespace: counter-net
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Domains whose registration expiry is monitored (infra/402_public_monitoring).
|
|
#
|
|
# Registration renewal is a manual act at the registrar, and losing a domain is
|
|
# not recoverable in the way losing a host is - so these are checked daily and
|
|
# alarm with two weeks of runway.
|
|
#
|
|
# root_domain is the estate's own domain; the rest are domains we own that are
|
|
# served from it or from a host in the inventory.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
monitored_domains:
|
|
- "{{ root_domain }}"
|
|
- arbret.com
|