counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
personal_infra/ansible/inventory.ini
counterweight c3a7ca0ba7
jar-lid-archive: a staging deployment on its own box 
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:56:30 +02:00

94 lines
3.9 KiB
INI
Raw Blame History

[vps]
vipy ansible_host=167.172.107.33 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
spacey ansible_host=64.227.112.128 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
monitoring ansible_host=64.226.70.190 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
[nodito_host]
nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# Requires the tailnet to be up on the control node.
[nodito_vms]
knots_box_local ansible_host=knots-box lan_ip=192.168.1.135 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
fulcrum_box_local ansible_host=fulcrum-box lan_ip=192.168.1.140 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
mempool_box_local ansible_host=mempool-box lan_ip=192.168.1.142 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
memos_box_local ansible_host=memos-box lan_ip=192.168.1.145 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
forgejo_runner_local ansible_host=forgejo-runner-box lan_ip=192.168.1.132 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.147 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
small_backups_local ansible_host=small-backups-box lan_ip=192.168.1.131 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# lan_ip read off nodito's ARP table for the VM's own MAC (bc:24:11:89:fc:1f),
# not guessed. Note .149 previously belonged to another machine - its stale host
# key was still in known_hosts and SSH refused the connection, which is the
# designed behaviour described in group_vars/nodito_vms.yml.
music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# lan_ip read from the Proxmox guest agent for VM 3300 (qm guest cmd
# network-get-interfaces), and the host key checked against the one the agent
# reads off the disk before it went into known_hosts.
jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1.151 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
[lapy]
localhost ansible_connection=local ansible_user=counterweight
[arbret]
prd-arbret ansible_host=167.99.242.62 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
[edge]
vipy
# The group is `observability`, NOT `monitoring` — there is a HOST named
# `monitoring` on line 5, and a group with the same name makes `hosts: monitoring`
# ambiguous. Ansible resolved it to the host and warned:
# [WARNING]: Found both group and host with same name: monitoring
[observability]
monitoring
[vpn_control]
spacey
[hypervisor]
nodito
[bitcoin]
knots_box_local
[electrum]
fulcrum_box_local
[mempool]
mempool_box_local
[monero]
monerod_box_local
[memos]
memos_box_local
[music]
music_box_local
# Staging only for now; a production host joins this group with its own host_vars.
[jar_lid_archive]
jar_lid_staging_local
[ci_runner]
forgejo_runner_local
[control]
localhost
# Every machine Ansible may configure as a server.
# Deliberately EXCLUDES [control] (your laptop) and [arbret].
[managed:children]
vps
nodito_host
nodito_vms
# Hosts that run Caddy and therefore have /etc/caddy/sites-enabled.
[caddy:children]
edge
observability
vpn_control
[backup_store]
small_backups_local
Reference in a new issue View git blame Copy permalink