counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
personal_infra/ansible/site.yml
counterweight c3a7ca0ba7
jar-lid-archive: a staging deployment on its own box 
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:56:30 +02:00

95 lines
5.8 KiB
YAML
Raw Blame History

---
# Everything, in the order it has to happen.
#
# This file is a TABLE OF CONTENTS, not a second source of truth. It says what
# runs and in what order. It does NOT say which hosts get what — that stays on
# the `hosts:` line inside each playbook, exactly where it is today. Nothing
# moves; this file only makes the set readable in one place.
#
# What runs on a host? ansible-playbook site.yml --limit <host> --list-hosts
# Who gets thing Y? the `hosts:` line in Y's own playbook
# What is a host? ansible-inventory --graph
#
# Run a slice with --limit, or run one playbook directly as before. Nothing here
# changes how any individual playbook behaves.
# ── Baseline: every managed machine ─────────────────────────────────────────
- import_playbook: infra/01_user_and_access_setup_playbook.yml
- import_playbook: infra/02_firewall_and_fail2ban_playbook.yml
- import_playbook: infra/900_install_rsync.yml
- import_playbook: infra/920_join_headscale_mesh.yml
# Idempotent and kept permanently: guarantees a rebuilt or restored host cannot
# quietly bring the Uptime-Kuma-era monitoring back.
- import_playbook: infra/409_remove_legacy_monitoring.yml
# ── Monitoring ──────────────────────────────────────────────────────────────
# Gatus first: the three plays below register endpoints with it, and registering
# against a host that is not serving yet would simply fail.
- import_playbook: services/gatus/deploy_gatus_playbook.yml
# The Signal transport for Gatus alerts. Shares a docker network with Gatus and
# publishes no ports - the API has no authentication. Needs a one-time manual
# device link; see roles/signal_api/README.md.
- import_playbook: services/signal-api/deploy_signal_api_playbook.yml
- import_playbook: infra/400_host_monitoring.yml
- import_playbook: infra/401_service_monitoring.yml
- import_playbook: infra/402_public_monitoring.yml
# Registers where the per-service probes report. The probes themselves are
# deployed by each service's own playbook further down; the endpoints must exist
# before the first push arrives.
- import_playbook: infra/403_service_probe_registration.yml
# 910_docker says `hosts: managed`, but only 5 of 11 managed hosts have or need
# Docker. Left out until it has a [docker] group — see the note in PLAN_7.
# ── The hypervisor ──────────────────────────────────────────────────────────
- import_playbook: infra/nodito/31_proxmox_community_repos_playbook.yml
- import_playbook: infra/nodito/32_zfs_pool_setup_playbook.yml
- import_playbook: infra/nodito/34_nut_ups_setup_playbook.yml
# ── Reverse proxy, before anything that registers a vhost ───────────────────
- import_playbook: services/caddy_playbook.yml
# ── Services ────────────────────────────────────────────────────────────────
- import_playbook: services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml
- import_playbook: services/fulcrum/deploy_fulcrum_playbook.yml
- import_playbook: services/datum-gateway/deploy_datum_gateway_playbook.yml
- import_playbook: services/mempool/deploy_mempool_playbook.yml
- import_playbook: services/monero/deploy_monero_playbook.yml
- import_playbook: services/xmrblocks/deploy_xmrblocks_playbook.yml
- import_playbook: services/memos/deploy_memos_playbook.yml
- import_playbook: services/navidrome/deploy_navidrome_playbook.yml
- import_playbook: services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml
- import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml
- import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml
- import_playbook: services/headscale/deploy_headscale_playbook.yml
- import_playbook: services/vaultwarden/deploy_vaultwarden_playbook.yml
- import_playbook: services/forgejo/deploy_forgejo_playbook.yml
- import_playbook: services/lnbits/deploy_lnbits_playbook.yml
- import_playbook: services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml
- import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml
# ── Backups: each source dumps itself, the box pulls ────────────────────────
- import_playbook: services/headscale/setup_backup_headscale.yml
- import_playbook: services/vaultwarden/setup_backup_vaultwarden.yml
- import_playbook: services/forgejo/setup_backup_forgejo.yml
- import_playbook: services/lnbits/setup_backup_lnbits.yml
- import_playbook: services/memos/setup_backup_memos.yml
- import_playbook: services/jar-lid-archive/setup_backup_jar_lid_archive.yml
- import_playbook: playbooks/backups.yml
# Deliberately not here. Every playbook in the repo is either imported above or
# listed below, so this file accounts for all of them:
#
# infra/910_docker_playbook.yml says `hosts: managed`, but Docker is on 5
# of 11 managed hosts and those 5 are exactly
# the ones that need it. Running it would
# install Docker on the Bitcoin node and the
# hypervisor. Needs a [docker] group first.
#
# infra/nodito/30_proxmox_bootstrap one-shot: bare-metal bootstrap, run once
# infra/nodito/33_..._cloud_template one-shot: builds the VM template
#
#
# services/vaultwarden/disable_ deliberate manual actions, not convergence
# vaultwarden_sign_ups_playbook.yml
# services/personal-blog/setup_
# deploy_alias_lapy.yml
Reference in a new issue View git blame Copy permalink