counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
138 commits 4 branches 1 tag 1.5 MiB
Find a file
counterweight c3a7ca0ba7
jar-lid-archive: a staging deployment on its own box 
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.

── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.

── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.

That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.

── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.

── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.

The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.

Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.

── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.

── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
  the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
  /health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:56:30 +02:00
ansible jar-lid-archive: a staging deployment on its own box 2026-10-03 23:56:30 +02:00
scripts navidrome: a music server on its own box, with beets in front of it 2026-09-27 11:09:40 +02:00
tofu/nodito jar-lid-archive: a staging deployment on its own box 2026-10-03 23:56:30 +02:00
.gitignore tofu: stop gitignoring the lock file and the VM inventory 2026-09-12 18:43:14 +02:00
01_infra_setup.md docs: mark Uptime Kuma as decommissioned 2026-09-11 22:43:56 +02:00
02_vps_core_services_setup.md docs: mark Uptime Kuma as decommissioned 2026-09-11 22:43:56 +02:00
03_vm_disk_enlargement.md little thingies 2026-03-22 21:25:08 +01:00
README.md docs: mark Uptime Kuma as decommissioned 2026-09-11 22:43:56 +02:00
requirements.txt uptime-kuma: annotate config and drop the unused collection 2026-09-11 22:43:56 +02:00

README.md

Personal infra

My repo documenting my personal infra, along with artifacts, scripts, etc.

How to use

Go through the different numbered markdowns in the repo root to do the different parts.

How to edit secrets

ansible-vault edit ansible/your_file_with_secrets.yml

Assumes that you've set ansible/.vault_pass with chmod 600.

Overview

Services

  • Reverse Proxy
    • Deployed on Vipy
    • Caddy
    • Plan install
    • File based config
    • Crossbackup to Desky via rsync
  • Uptime Kuma — decommissioned 2026-09-11, see archive/uptime_kuma/
    • Deployed on Vipy
    • Crossbackup to Desky via rsync
  • Vaultwarden
    • Deployed on Desky
    • Crossbackup to Vipy via rsync
  • Gitea
    • Deployed on Desky
    • Crossbackup to Vipy via rsync
  • Immich
    • Deployed on Desky
  • VPN
    • All set up on Vipy
  • Bitcoin Knots
    • Deployed on Desky
  • electrs
  • Synapse Server
  • Phoenix D + LNBits
  • Backups

Infra

  • Laptop (Lapy)
  • One beefy desktop (Desky)
  • One VPS (Vipy)