Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box (nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz through vipy's Caddy. New role roles/jar_lid_archive; per-host values in host_vars, so a production host is a new [jar_lid_archive] entry, not a new playbook. ── Containers, following :master ────────────────────────────────────────── The same three services as the app repo's compose.yaml, so staging rehearses what dev and CI run. Every playbook run pulls :master and `up -d --wait`s; compose recreates web only when the digest moved. --wait gates on web's /health/, which checks the database and the bucket, so a green run means the app serves. Upgrading is re-running the playbook. ── web is on the host network ───────────────────────────────────────────── Docker's published ports are DNATed before UFW sees them, so `8000:8000` would answer on the LAN whatever UFW says, and binding to the tailnet IP races tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly :8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy gets 200. That matters for HTTPS: the app trusts X-Forwarded-Proto (DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client. Login POST through the public URL passes CSRF; cookies are Secure. ── Secrets ──────────────────────────────────────────────────────────────── Generated straight into host_vars/jar_lid_staging_local/vault.yml, never printed: Django key, Postgres and S3 secrets, and seed passwords for the fixture's alice/bob/admin usernames - never the fixture's published passwords, since the site is public. The group vault gains one line, this host's Gatus push token; decrypted, it differs from before by exactly that line. ── Backups, and the outage the first one caused ─────────────────────────── One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end to end, including decrypting a pulled artefact on lapy. The first test run stopped web AND storage and left the app down 26 minutes: SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it unhealthy, and web's depends_on never started it. Now only web is stopped - it is the only writer to both stores, so they are still captured at one instant - and the healthchecks have start_periods sized for the disk. Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and chowns it to seaweed, so on a RESTART of the same container root cannot re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here with rm -f first; the app repo needs the same line. ── The disk is the problem ──────────────────────────────────────────────── The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads (~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs (NVMe) is the fix; noted in the README, not done. ── Also ─────────────────────────────────────────────────────────────────── - lan_ip read from the Proxmox guest agent; host key checked against the one the agent reads off the disk before it went into known_hosts. - Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public /health/ + DNS (402), backup-dump/backup-store. - tofu: the VM definition. - Deployed v2026.10.03.1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| .terraform.lock.hcl | ||
| main.tf | ||
| provider.tf | ||
| README.md | ||
| terraform.tfvars.example | ||
| variables.tf | ||
| versions.tf | ||
| vms.auto.tfvars | ||
Nodito VMs with OpenTofu (Proxmox)
This directory lets you declare VMs on the nodito Proxmox node and apply with OpenTofu. It clones the Ansible-built template debian-13-cloud-init and places disks on the ZFS pool proxmox-tank-1.
Prereqs
- Proxmox API token with VM privileges. Example: user
root@pam, token nametofu. - OpenTofu installed.
sudo apt-get update sudo apt-get install -y apt-transport-https ca-certificates curl gnupg sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://get.opentofu.org/opentofu.gpg | sudo tee /etc/apt/keyrings/opentofu.gpg >/dev/null curl -fsSL https://packages.opentofu.org/opentofu/tofu/gpgkey | sudo gpg --no-tty --batch --dearmor -o /etc/apt/keyrings/opentofu-repo.gpg >/dev/null sudo chmod a+r /etc/apt/keyrings/opentofu.gpg /etc/apt/keyrings/opentofu-repo.gpg echo \ "deb [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main deb-src [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main" | \ sudo tee /etc/apt/sources.list.d/opentofu.list > /dev/null sudo chmod a+r /etc/apt/sources.list.d/opentofu.list sudo apt-get update sudo apt-get install -y tofu tofu version - The Ansible template exists:
debian-13-cloud-init(VMID 9001 by default).
Provider Auth
Credentials are the only thing not in git. Copy terraform.tfvars.example to
terraform.tfvars (gitignored) and set:
proxmox_api_url(e.g.https://nodito:8006/api2/json)proxmox_api_token_id(e.g.root@pam!tofu)proxmox_api_token_secret
Alternatively, export them as TF_VAR_proxmox_api_token_secret etc.
Declare VMs
VMs are declared in vms.auto.tfvars, which is committed. *.auto.tfvars is
loaded automatically, so it needs no -var-file. Example entry:
vms = {
web1 = {
name = "web1"
cores = 2
memory_mb = 2048
disk_size_gb = 20
ipconfig0 = "ip=dhcp" # or "ip=192.168.1.50/24,gw=192.168.1.1"
}
}
All VM disks are created on zfs_storage_name (defaults to proxmox-tank-1). Network attaches to vmbr0. VLAN can be set per-VM with vlan_tag.
Usage
tofu init
tofu plan
tofu apply
terraform.tfvars and vms.auto.tfvars are both auto-loaded.
VMs are created once and then protected: the module sets
lifecycle.prevent_destroy = trueand ignores subsequent config changes. After the initial apply, manage day‑2 changes directly in Proxmox (or remove the lifecycle block if you need OpenTofu to own ongoing updates).
Notes
- Clones are full clones by default (
full_clone = true). - Cloud-init injects
cloud_init_userandssh_authorized_keys. .terraform.lock.hclis committed: it pins the provider hashes, which matters becauseversions.tftracks a release candidate (3.0.2-rc05).- State is local (
terraform.tfstate, gitignored) and has no remote backend, so it exists only on the machine that last rantofu apply. - The map is not a complete inventory of nodito: state also manages
bastion-box(1100) andnonkeiwaisi-box(3300), which are not declared invms.auto.tfvars.tofu planis clean today, but relaxing thelifecycleblock without first declaring them would put them up for destruction. - Disks use
scsi0on ZFS withdiscardenabled.