2026-09-11 21:32:23 +02:00
|
|
|
[vps]
|
2026-10-19 22:10:22 +02:00
|
|
|
vipy ansible_host=167.172.98.204 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
|
|
|
|
spacey ansible_host=46.101.250.26 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
2026-09-12 15:14:47 +02:00
|
|
|
|
2026-09-11 21:32:23 +02:00
|
|
|
[nodito_host]
|
|
|
|
|
nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
|
|
|
|
|
2026-09-12 15:14:47 +02:00
|
|
|
# Requires the tailnet to be up on the control node.
|
2026-09-11 21:32:23 +02:00
|
|
|
[nodito_vms]
|
2026-09-12 15:14:47 +02:00
|
|
|
knots_box_local ansible_host=knots-box lan_ip=192.168.1.135 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
|
|
|
|
fulcrum_box_local ansible_host=fulcrum-box lan_ip=192.168.1.140 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
|
|
|
|
mempool_box_local ansible_host=mempool-box lan_ip=192.168.1.142 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
2026-10-19 22:10:22 +02:00
|
|
|
memos_box_local ansible_host=memos-box lan_ip=192.168.1.130 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
2026-09-12 15:14:47 +02:00
|
|
|
forgejo_runner_local ansible_host=forgejo-runner-box lan_ip=192.168.1.132 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
2026-10-19 22:10:22 +02:00
|
|
|
arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.145 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
2026-09-12 15:14:47 +02:00
|
|
|
small_backups_local ansible_host=small-backups-box lan_ip=192.168.1.131 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
monero: a full node on its own box, reachable from Feather over the WAN
monerod v0.18.5.1 on monerod-box (VMID 2400), Debian 13, systemd, no Docker.
Syncing at 24% as of this commit, 0 restarts, 0 OOM kills.
── The build-from-source requirement could not be met ──────────────────────
The intent was to build it ourselves and check the result against the published
checksums. That is not achievable for this release, and the role's README says
so at length so nobody re-derives it:
contrib/guix - the reproducible build that yields a byte-identical tarball -
exists ONLY on monero master. At tag v0.18.5.1 contrib/ contains `gitian`,
whose README wants an Ubuntu 18.04 host (EOL 2023) and python-cheetah.
A plain `make release-static` is not reproducible at all, so it produces a
binary whose hash matches nothing - failing both halves of the ask.
So we install the published binary and prove its provenance instead:
1. binaryFate's key is VENDORED at files/binaryfate.asc. A trust root fetched
over the network at the moment you need it is not a trust root. Its
fingerprint is asserted after import regardless.
2. hashes.txt is verified, and the assertion is on gpg's `VALIDSIG <fpr>`
status line - NOT on the string "Good signature", which is what
bitcoin_knots greps for and which passes for any key in the keyring.
3. The hash is looked up in the output of `gpg --decrypt`, not in the raw
clearsigned file, so nothing unsigned is ever searched.
4. The lookup requires hash AND filename on one line.
Tested with a positive and a negative control before deploying: the real
tarball matches, a corrupted hash correctly fails. monero_sha256 in defaults/
is a second in-git assertion so a version bump cannot happen without the
checksum moving in the same diff.
When v0.19 ships with guix on the release branch this gets easier, and the
README records why: the published monero-linux-x64 tarball unpacks to
monero-x86_64-linux-gnu-v0.18.5.1/, i.e. it IS the guix DISTNAME renamed, so
sha256(guix output) will equal the hashes.txt entry exactly.
── The 500 GB disk is provisioned in code ──────────────────────────────────
knots-box mounts /mnt/knots_data (~875 GB of chain) entirely by hand; that
mount appears nowhere in Ansible and a rebuild of that host would come up with
the blockchain silently missing. tasks/storage.yml does not repeat it.
Whole device, no partition table. Everything after mkfs goes by UUID, because
/dev/sdX is assigned by the kernel at boot and this estate has been bitten
eight times by an identifier written down once whose thing then moved.
monero_data_device is used ONLY to create the filesystem. Guards refuse to
touch the root disk or to overwrite an unrecognised filesystem, and a final
assert confirms the data directory is not still on /.
fstab gets `nofail` so a missing disk cannot drop a headless VM into an
emergency shell - and the unit gets `RequiresMountsFor=`, without which monerod
would then start, find an empty directory, and write a 300 GB chain onto a
10 GB root disk. With it the unit refuses to start and infra/401 says so within
five minutes.
The mount-point task and the ownership task deliberately share owner and mode.
Writing the obvious root:root 0755 on the first made them disagree and every
run flipped the directory back and forth; caught by re-running --check.
── Exposure: one HTTPS vhost, zero new open ports ──────────────────────────
Two RPC servers, not one: unrestricted on 127.0.0.1:18081 for administration
and the health check, restricted on :18089 for everything else. That is
`rpc-restricted-bind-port` and deliberately NOT `restricted-rpc=1`, which would
restrict the loopback server too. Verified live: the restricted port reports
restricted=true and blanks version and start_time.
--rpc-login is HTTP Digest (contrib/epee/src/http_auth.cpp) and
core_rpc_server::init hands the same login to both servers. Caddy on vipy does
TLS and routing only - no caddy_site_basic_auth, because a Basic challenge
would collide with the Digest challenge Feather answers. Verified end to end
from the WAN: 401 unauthenticated, get_block_count over /json_rpc when
authenticated, 401 again with a wrong password.
No UFW rule, and that is correct: tailnet traffic bypasses UFW entirely because
`-A INPUT -j ts-input` sits ahead of every ufw-* chain. Confirmed on mempool-box
before relying on it here.
P2P is outbound only. Nothing forwards 18080, so the node dials out, syncs and
serves us without relaying for strangers; no-igd follows from that.
── Monitoring: infra, systemd, public. No backups ──────────────────────────
Six Gatus endpoints, all green: liveness, disk, services/monerod-box-local/
monerod, probe/monerod, dns/monero.contrapeso.xyz, public/monero.
The probe does NOT assert `synchronized`. A full sync takes days and a check
requiring it would report DOWN for every one of them - an alarm with the
batteries taken out. Height and sync state go in the message instead.
The timer carries no `Requires=monerod.service`. Fulcrum's did, and sat
active+enabled for seven months without firing once. public/monero asserts 401,
not 200, so it goes red if rpc-login ever stops being enforced.
No backups, deliberately, following the phoenixd precedent: the chain is
re-syncable public data and there is no wallet on this node.
── Known constraint ────────────────────────────────────────────────────────
The VM has 2 GB and no swap; monerod's RSS is ~1.3 GB mid-sync. It is holding
(0 OOM kills, ~1 GB available, kernel reclaiming cache) but 8 GB is the
recommendation once the initial sync is done. out-peers/in-peers are held at
16/8 partly for this reason. Bumping it is a Proxmox resize plus memory_mb
here, since the vm resource sets ignore_changes = all.
The 500 GB data disk is likewise invisible to tofu - it was attached in the UI
and the vm resource ignores it - so vms.auto.tfvars now carries a comment
saying where it is and what mounts it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 17:50:46 +02:00
|
|
|
monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
navidrome: a music server on its own box, with beets in front of it
Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.
── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.
Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.
The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.
A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.
── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.
── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.
/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.
── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).
/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.
── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".
inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.
50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.
No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 11:09:40 +02:00
|
|
|
# lan_ip read off nodito's ARP table for the VM's own MAC (bc:24:11:89:fc:1f),
|
|
|
|
|
# not guessed. Note .149 previously belonged to another machine - its stale host
|
|
|
|
|
# key was still in known_hosts and SSH refused the connection, which is the
|
|
|
|
|
# designed behaviour described in group_vars/nodito_vms.yml.
|
|
|
|
|
music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
jar-lid-archive: a staging deployment on its own box
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.
── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.
── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.
That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.
── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.
── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.
The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.
Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.
── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.
── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
/health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:56:30 +02:00
|
|
|
# lan_ip read from the Proxmox guest agent for VM 3300 (qm guest cmd
|
|
|
|
|
# network-get-interfaces), and the host key checked against the one the agent
|
|
|
|
|
# reads off the disk before it went into known_hosts.
|
|
|
|
|
jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1.151 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
|
2026-09-11 21:32:23 +02:00
|
|
|
|
|
|
|
|
# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
|
|
|
|
|
[lapy]
|
2026-09-12 17:16:26 +02:00
|
|
|
localhost ansible_connection=local ansible_user=counterweight
|
2026-09-11 21:32:23 +02:00
|
|
|
|
2026-09-11 21:51:50 +02:00
|
|
|
[edge]
|
|
|
|
|
vipy
|
|
|
|
|
|
2026-10-19 22:10:22 +02:00
|
|
|
# Gatus and its Signal transport. Shares spacey with headscale since the
|
|
|
|
|
# 2026-10-17 rebuild; the separate `monitoring` droplet is gone.
|
ansible: add site.yml, and rename the monitoring group off the host's name
site.yml is a TABLE OF CONTENTS, not a second source of truth. It is 25
import_playbook: lines and comments - no `hosts:`, no `roles:`. Which hosts get
what stays on the `hosts:` line inside each playbook, exactly where it already
was; nothing moved. Every role is already wrapped in a thin playbook carrying
its own `hosts:` line, so there is no roles-vs-playbooks split to reconcile:
from here everything is a playbook.
What it buys:
What runs on a host? ansible-playbook site.yml --limit <host> --list-hosts
Who gets thing Y? the `hosts:` line in Y's own playbook
What is a host? ansible-inventory --graph
Note --list-hosts, not --list-tasks: the latter prints every play regardless of
--limit, so it will happily show you the bitcoin play under memos-box.
Nine playbooks are deliberately excluded and the file names every one with a
reason, so it accounts for all of them: the three infra/4xx monitoring plays
(still assert on the removed Uptime Kuma credentials and fail immediately),
910_docker (says `hosts: managed`, but Docker is on 5 of 11 managed hosts and
those 5 are exactly the ones that need it - running it installs Docker on the
Bitcoin node and the hypervisor), two nodito one-shots, the Kuma notification
setup, and two deliberate manual actions.
Writing it surfaced an inventory collision. There is a HOST named `monitoring`
in [vps] AND a group [monitoring], so Ansible warned and resolved `hosts:
monitoring` to the host:
[WARNING]: Found both group and host with same name: monitoring
The group is renamed to [observability]; the host keeps its name. [caddy:children]
and the two ntfy playbooks follow. Behaviour is unchanged - `hosts: monitoring`
already resolved to the host - but the ambiguity is gone and the warning with it.
Verified: inventory graph is warning-free, site.yml passes --syntax-check, and
per-host play counts are identical before and after the rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 21:24:09 +02:00
|
|
|
[observability]
|
2026-10-19 22:10:22 +02:00
|
|
|
spacey
|
2026-09-11 21:51:50 +02:00
|
|
|
|
|
|
|
|
[vpn_control]
|
|
|
|
|
spacey
|
|
|
|
|
|
|
|
|
|
[hypervisor]
|
|
|
|
|
nodito
|
|
|
|
|
|
|
|
|
|
[bitcoin]
|
|
|
|
|
knots_box_local
|
|
|
|
|
|
|
|
|
|
[electrum]
|
|
|
|
|
fulcrum_box_local
|
|
|
|
|
|
|
|
|
|
[mempool]
|
|
|
|
|
mempool_box_local
|
|
|
|
|
|
monero: a full node on its own box, reachable from Feather over the WAN
monerod v0.18.5.1 on monerod-box (VMID 2400), Debian 13, systemd, no Docker.
Syncing at 24% as of this commit, 0 restarts, 0 OOM kills.
── The build-from-source requirement could not be met ──────────────────────
The intent was to build it ourselves and check the result against the published
checksums. That is not achievable for this release, and the role's README says
so at length so nobody re-derives it:
contrib/guix - the reproducible build that yields a byte-identical tarball -
exists ONLY on monero master. At tag v0.18.5.1 contrib/ contains `gitian`,
whose README wants an Ubuntu 18.04 host (EOL 2023) and python-cheetah.
A plain `make release-static` is not reproducible at all, so it produces a
binary whose hash matches nothing - failing both halves of the ask.
So we install the published binary and prove its provenance instead:
1. binaryFate's key is VENDORED at files/binaryfate.asc. A trust root fetched
over the network at the moment you need it is not a trust root. Its
fingerprint is asserted after import regardless.
2. hashes.txt is verified, and the assertion is on gpg's `VALIDSIG <fpr>`
status line - NOT on the string "Good signature", which is what
bitcoin_knots greps for and which passes for any key in the keyring.
3. The hash is looked up in the output of `gpg --decrypt`, not in the raw
clearsigned file, so nothing unsigned is ever searched.
4. The lookup requires hash AND filename on one line.
Tested with a positive and a negative control before deploying: the real
tarball matches, a corrupted hash correctly fails. monero_sha256 in defaults/
is a second in-git assertion so a version bump cannot happen without the
checksum moving in the same diff.
When v0.19 ships with guix on the release branch this gets easier, and the
README records why: the published monero-linux-x64 tarball unpacks to
monero-x86_64-linux-gnu-v0.18.5.1/, i.e. it IS the guix DISTNAME renamed, so
sha256(guix output) will equal the hashes.txt entry exactly.
── The 500 GB disk is provisioned in code ──────────────────────────────────
knots-box mounts /mnt/knots_data (~875 GB of chain) entirely by hand; that
mount appears nowhere in Ansible and a rebuild of that host would come up with
the blockchain silently missing. tasks/storage.yml does not repeat it.
Whole device, no partition table. Everything after mkfs goes by UUID, because
/dev/sdX is assigned by the kernel at boot and this estate has been bitten
eight times by an identifier written down once whose thing then moved.
monero_data_device is used ONLY to create the filesystem. Guards refuse to
touch the root disk or to overwrite an unrecognised filesystem, and a final
assert confirms the data directory is not still on /.
fstab gets `nofail` so a missing disk cannot drop a headless VM into an
emergency shell - and the unit gets `RequiresMountsFor=`, without which monerod
would then start, find an empty directory, and write a 300 GB chain onto a
10 GB root disk. With it the unit refuses to start and infra/401 says so within
five minutes.
The mount-point task and the ownership task deliberately share owner and mode.
Writing the obvious root:root 0755 on the first made them disagree and every
run flipped the directory back and forth; caught by re-running --check.
── Exposure: one HTTPS vhost, zero new open ports ──────────────────────────
Two RPC servers, not one: unrestricted on 127.0.0.1:18081 for administration
and the health check, restricted on :18089 for everything else. That is
`rpc-restricted-bind-port` and deliberately NOT `restricted-rpc=1`, which would
restrict the loopback server too. Verified live: the restricted port reports
restricted=true and blanks version and start_time.
--rpc-login is HTTP Digest (contrib/epee/src/http_auth.cpp) and
core_rpc_server::init hands the same login to both servers. Caddy on vipy does
TLS and routing only - no caddy_site_basic_auth, because a Basic challenge
would collide with the Digest challenge Feather answers. Verified end to end
from the WAN: 401 unauthenticated, get_block_count over /json_rpc when
authenticated, 401 again with a wrong password.
No UFW rule, and that is correct: tailnet traffic bypasses UFW entirely because
`-A INPUT -j ts-input` sits ahead of every ufw-* chain. Confirmed on mempool-box
before relying on it here.
P2P is outbound only. Nothing forwards 18080, so the node dials out, syncs and
serves us without relaying for strangers; no-igd follows from that.
── Monitoring: infra, systemd, public. No backups ──────────────────────────
Six Gatus endpoints, all green: liveness, disk, services/monerod-box-local/
monerod, probe/monerod, dns/monero.contrapeso.xyz, public/monero.
The probe does NOT assert `synchronized`. A full sync takes days and a check
requiring it would report DOWN for every one of them - an alarm with the
batteries taken out. Height and sync state go in the message instead.
The timer carries no `Requires=monerod.service`. Fulcrum's did, and sat
active+enabled for seven months without firing once. public/monero asserts 401,
not 200, so it goes red if rpc-login ever stops being enforced.
No backups, deliberately, following the phoenixd precedent: the chain is
re-syncable public data and there is no wallet on this node.
── Known constraint ────────────────────────────────────────────────────────
The VM has 2 GB and no swap; monerod's RSS is ~1.3 GB mid-sync. It is holding
(0 OOM kills, ~1 GB available, kernel reclaiming cache) but 8 GB is the
recommendation once the initial sync is done. out-peers/in-peers are held at
16/8 partly for this reason. Bumping it is a Proxmox resize plus memory_mb
here, since the vm resource sets ignore_changes = all.
The 500 GB data disk is likewise invisible to tofu - it was attached in the UI
and the vm resource ignores it - so vms.auto.tfvars now carries a comment
saying where it is and what mounts it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 17:50:46 +02:00
|
|
|
[monero]
|
|
|
|
|
monerod_box_local
|
|
|
|
|
|
2026-09-11 21:51:50 +02:00
|
|
|
[memos]
|
|
|
|
|
memos_box_local
|
|
|
|
|
|
navidrome: a music server on its own box, with beets in front of it
Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.
── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.
Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.
The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.
A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.
── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.
── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.
/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.
── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).
/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.
── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".
inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.
50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.
No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 11:09:40 +02:00
|
|
|
[music]
|
|
|
|
|
music_box_local
|
|
|
|
|
|
jar-lid-archive: a staging deployment on its own box
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.
── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.
── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.
That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.
── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.
── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.
The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.
Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.
── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.
── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
/health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:56:30 +02:00
|
|
|
# Staging only for now; a production host joins this group with its own host_vars.
|
|
|
|
|
[jar_lid_archive]
|
|
|
|
|
jar_lid_staging_local
|
|
|
|
|
|
2026-09-11 21:51:50 +02:00
|
|
|
[ci_runner]
|
|
|
|
|
forgejo_runner_local
|
|
|
|
|
|
|
|
|
|
[control]
|
|
|
|
|
localhost
|
|
|
|
|
|
|
|
|
|
# Every machine Ansible may configure as a server.
|
2026-10-19 22:10:22 +02:00
|
|
|
# Deliberately EXCLUDES [control] (your laptop).
|
2026-09-11 21:51:50 +02:00
|
|
|
[managed:children]
|
|
|
|
|
vps
|
|
|
|
|
nodito_host
|
2026-09-11 23:03:25 +02:00
|
|
|
nodito_vms
|
|
|
|
|
|
|
|
|
|
# Hosts that run Caddy and therefore have /etc/caddy/sites-enabled.
|
|
|
|
|
[caddy:children]
|
|
|
|
|
edge
|
ansible: add site.yml, and rename the monitoring group off the host's name
site.yml is a TABLE OF CONTENTS, not a second source of truth. It is 25
import_playbook: lines and comments - no `hosts:`, no `roles:`. Which hosts get
what stays on the `hosts:` line inside each playbook, exactly where it already
was; nothing moved. Every role is already wrapped in a thin playbook carrying
its own `hosts:` line, so there is no roles-vs-playbooks split to reconcile:
from here everything is a playbook.
What it buys:
What runs on a host? ansible-playbook site.yml --limit <host> --list-hosts
Who gets thing Y? the `hosts:` line in Y's own playbook
What is a host? ansible-inventory --graph
Note --list-hosts, not --list-tasks: the latter prints every play regardless of
--limit, so it will happily show you the bitcoin play under memos-box.
Nine playbooks are deliberately excluded and the file names every one with a
reason, so it accounts for all of them: the three infra/4xx monitoring plays
(still assert on the removed Uptime Kuma credentials and fail immediately),
910_docker (says `hosts: managed`, but Docker is on 5 of 11 managed hosts and
those 5 are exactly the ones that need it - running it installs Docker on the
Bitcoin node and the hypervisor), two nodito one-shots, the Kuma notification
setup, and two deliberate manual actions.
Writing it surfaced an inventory collision. There is a HOST named `monitoring`
in [vps] AND a group [monitoring], so Ansible warned and resolved `hosts:
monitoring` to the host:
[WARNING]: Found both group and host with same name: monitoring
The group is renamed to [observability]; the host keeps its name. [caddy:children]
and the two ntfy playbooks follow. Behaviour is unchanged - `hosts: monitoring`
already resolved to the host - but the ambiguity is gone and the warning with it.
Verified: inventory graph is warning-free, site.yml passes --syntax-check, and
per-host play counts are identical before and after the rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 21:24:09 +02:00
|
|
|
observability
|
2026-09-11 23:03:25 +02:00
|
|
|
vpn_control
|
2026-09-12 15:14:47 +02:00
|
|
|
|
|
|
|
|
[backup_store]
|
|
|
|
|
small_backups_local
|