counterweight
c3a7ca0ba7
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
144 lines
3.6 KiB
Django/Jinja
144 lines
3.6 KiB
Django/Jinja
{#
|
|
Headscale ACL policy.
|
|
|
|
CAPTURED FROM THE LIVE FILE on spacey, 2026-09-19. Until then this playbook
|
|
wrote `{}` here - an empty policy - while the real rules were maintained by
|
|
hand on the box. Ansible had already flattened them once:
|
|
/etc/headscale/acl.json.937464.2026-09-12@14:03:08~ is its own backup of a
|
|
1596-byte policy it replaced with {}. The next full site.yml run would have
|
|
done it again, taking the whole tailnet's access rules with it.
|
|
|
|
So this is the source of truth now. It is a template and not inline `content:`
|
|
(the style used elsewhere in this playbook) precisely because it needs to be
|
|
diffable in review - an empty policy is the kind of change that must never
|
|
again slip through as three invisible characters.
|
|
|
|
ADDING A HOST: it needs a tagOwners entry, and an acls rule naming the exact
|
|
ports its peers may reach. A node with no rule is INVISIBLE to the others -
|
|
it will not appear in `tailscale status` and MagicDNS will not resolve it,
|
|
which surfaces as a Caddy 502 with "no such host" rather than as a refused
|
|
connection.
|
|
#}
|
|
{
|
|
"acls": [
|
|
{
|
|
"action": "accept",
|
|
"dst": [
|
|
"*:22",
|
|
"tag:arbret-staging-box:4117",
|
|
"tag:prd-arbret:4117",
|
|
"tag:nodito:8006"
|
|
],
|
|
"src": [
|
|
"tag:framie"
|
|
]
|
|
},
|
|
{
|
|
"action": "accept",
|
|
"dst": [
|
|
"tag:knots-box:8333",
|
|
"tag:knots-box:7152",
|
|
"tag:knots-box:23334",
|
|
"tag:fulcrum-box:50002",
|
|
"tag:mempool-box:8080",
|
|
"tag:memos-box:5230",
|
|
"tag:music-box:4533",
|
|
"tag:jar-lid-archive-staging-box:8000",
|
|
"tag:monerod-box:18089",
|
|
"tag:monerod-box:8081",
|
|
"tag:arbret-staging-box:80",
|
|
"tag:stalwart-box:25",
|
|
"tag:stalwart-box:465",
|
|
"tag:stalwart-box:993",
|
|
"tag:stalwart-box:4190",
|
|
"tag:stalwart-box:8080"
|
|
],
|
|
"src": [
|
|
"tag:prd-vipy"
|
|
]
|
|
},
|
|
{
|
|
"action": "accept",
|
|
"dst": [
|
|
"tag:knots-box:8332",
|
|
"tag:knots-box:28332-28335"
|
|
],
|
|
"src": [
|
|
"tag:fulcrum-box"
|
|
]
|
|
},
|
|
{
|
|
"action": "accept",
|
|
"dst": [
|
|
"tag:knots-box:8332",
|
|
"tag:fulcrum-box:50001"
|
|
],
|
|
"src": [
|
|
"tag:mempool-box"
|
|
]
|
|
},
|
|
{
|
|
"action": "accept",
|
|
"dst": [
|
|
"tag:prd-arbret:22",
|
|
"tag:prd-vipy:22",
|
|
"tag:memos-box:22",
|
|
"tag:stalwart-box:22",
|
|
"tag:jar-lid-archive-staging-box:22"
|
|
],
|
|
"src": [
|
|
"tag:small-backups-box"
|
|
]
|
|
}
|
|
],
|
|
"tagOwners": {
|
|
"tag:arbret-staging-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:forgejo-runner-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:framie": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:fulcrum-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:jar-lid-archive-staging-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:knots-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:memos-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:mempool-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:monerod-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:music-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:nodito": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:prd-arbret": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:prd-monitoring": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:prd-vipy": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:small-backups-box": [
|
|
"{{ headscale_namespace }}@"
|
|
],
|
|
"tag:stalwart-box": [
|
|
"{{ headscale_namespace }}@"
|
|
]
|
|
}
|
|
}
|