Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.
── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.
── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.
That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.
── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.
── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.
The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.
Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.
── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.
── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
/health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.
── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.
Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.
The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.
A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.
── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.
── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.
/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.
── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).
/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.
── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".
inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.
50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.
No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
monerod v0.18.5.1 on monerod-box (VMID 2400), Debian 13, systemd, no Docker.
Syncing at 24% as of this commit, 0 restarts, 0 OOM kills.
── The build-from-source requirement could not be met ──────────────────────
The intent was to build it ourselves and check the result against the published
checksums. That is not achievable for this release, and the role's README says
so at length so nobody re-derives it:
contrib/guix - the reproducible build that yields a byte-identical tarball -
exists ONLY on monero master. At tag v0.18.5.1 contrib/ contains `gitian`,
whose README wants an Ubuntu 18.04 host (EOL 2023) and python-cheetah.
A plain `make release-static` is not reproducible at all, so it produces a
binary whose hash matches nothing - failing both halves of the ask.
So we install the published binary and prove its provenance instead:
1. binaryFate's key is VENDORED at files/binaryfate.asc. A trust root fetched
over the network at the moment you need it is not a trust root. Its
fingerprint is asserted after import regardless.
2. hashes.txt is verified, and the assertion is on gpg's `VALIDSIG <fpr>`
status line - NOT on the string "Good signature", which is what
bitcoin_knots greps for and which passes for any key in the keyring.
3. The hash is looked up in the output of `gpg --decrypt`, not in the raw
clearsigned file, so nothing unsigned is ever searched.
4. The lookup requires hash AND filename on one line.
Tested with a positive and a negative control before deploying: the real
tarball matches, a corrupted hash correctly fails. monero_sha256 in defaults/
is a second in-git assertion so a version bump cannot happen without the
checksum moving in the same diff.
When v0.19 ships with guix on the release branch this gets easier, and the
README records why: the published monero-linux-x64 tarball unpacks to
monero-x86_64-linux-gnu-v0.18.5.1/, i.e. it IS the guix DISTNAME renamed, so
sha256(guix output) will equal the hashes.txt entry exactly.
── The 500 GB disk is provisioned in code ──────────────────────────────────
knots-box mounts /mnt/knots_data (~875 GB of chain) entirely by hand; that
mount appears nowhere in Ansible and a rebuild of that host would come up with
the blockchain silently missing. tasks/storage.yml does not repeat it.
Whole device, no partition table. Everything after mkfs goes by UUID, because
/dev/sdX is assigned by the kernel at boot and this estate has been bitten
eight times by an identifier written down once whose thing then moved.
monero_data_device is used ONLY to create the filesystem. Guards refuse to
touch the root disk or to overwrite an unrecognised filesystem, and a final
assert confirms the data directory is not still on /.
fstab gets `nofail` so a missing disk cannot drop a headless VM into an
emergency shell - and the unit gets `RequiresMountsFor=`, without which monerod
would then start, find an empty directory, and write a 300 GB chain onto a
10 GB root disk. With it the unit refuses to start and infra/401 says so within
five minutes.
The mount-point task and the ownership task deliberately share owner and mode.
Writing the obvious root:root 0755 on the first made them disagree and every
run flipped the directory back and forth; caught by re-running --check.
── Exposure: one HTTPS vhost, zero new open ports ──────────────────────────
Two RPC servers, not one: unrestricted on 127.0.0.1:18081 for administration
and the health check, restricted on :18089 for everything else. That is
`rpc-restricted-bind-port` and deliberately NOT `restricted-rpc=1`, which would
restrict the loopback server too. Verified live: the restricted port reports
restricted=true and blanks version and start_time.
--rpc-login is HTTP Digest (contrib/epee/src/http_auth.cpp) and
core_rpc_server::init hands the same login to both servers. Caddy on vipy does
TLS and routing only - no caddy_site_basic_auth, because a Basic challenge
would collide with the Digest challenge Feather answers. Verified end to end
from the WAN: 401 unauthenticated, get_block_count over /json_rpc when
authenticated, 401 again with a wrong password.
No UFW rule, and that is correct: tailnet traffic bypasses UFW entirely because
`-A INPUT -j ts-input` sits ahead of every ufw-* chain. Confirmed on mempool-box
before relying on it here.
P2P is outbound only. Nothing forwards 18080, so the node dials out, syncs and
serves us without relaying for strangers; no-igd follows from that.
── Monitoring: infra, systemd, public. No backups ──────────────────────────
Six Gatus endpoints, all green: liveness, disk, services/monerod-box-local/
monerod, probe/monerod, dns/monero.contrapeso.xyz, public/monero.
The probe does NOT assert `synchronized`. A full sync takes days and a check
requiring it would report DOWN for every one of them - an alarm with the
batteries taken out. Height and sync state go in the message instead.
The timer carries no `Requires=monerod.service`. Fulcrum's did, and sat
active+enabled for seven months without firing once. public/monero asserts 401,
not 200, so it goes red if rpc-login ever stops being enforced.
No backups, deliberately, following the phoenixd precedent: the chain is
re-syncable public data and there is no wallet on this node.
── Known constraint ────────────────────────────────────────────────────────
The VM has 2 GB and no swap; monerod's RSS is ~1.3 GB mid-sync. It is holding
(0 OOM kills, ~1 GB available, kernel reclaiming cache) but 8 GB is the
recommendation once the initial sync is done. out-peers/in-peers are held at
16/8 partly for this reason. Bumping it is a Proxmox resize plus memory_mb
here, since the vm resource sets ignore_changes = all.
The 500 GB data disk is likewise invisible to tofu - it was attached in the UI
and the vm resource ignores it - so vms.auto.tfvars now carries a comment
saying where it is and what mounts it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes: detection windows tightened, a Signal transport deployed, and
alerts attached to all 84 non-transport endpoints.
── Failing faster ──────────────────────────────────────────────────────────
The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks
"did anything arrive in the last interval", so real detection is 1-2x the
window. And a window can only ever be as tight as the push frequency - which is
why two checks moved rather than just having their numbers changed.
liveness, cpu, ups, service-health, probes 16m -> 11m
disk, zfs daily/30h -> 6-hourly/7h
backup store + pull job daily/30h -> 6-hourly/7h
DNS records 24h -> 6h
backup dump 30h -> 26h
backup-dump stays slow because the dump genuinely is daily. The store-side check
catches the same fault within 6h by reading the source's dump timestamp out of
the artefact filename, so 26h is a backstop rather than the primary signal.
── Thresholds differ by check type, deliberately ───────────────────────────
`failure-threshold` counts consecutive failures, but "consecutive" is a
different amount of wall-clock time per check: a push endpoint produces one
failure per heartbeat window, a pulled one per interval. The default of 3 would
mean 33 minutes on an 11m heartbeat and over a day on a 7h one.
More importantly the heartbeat window ALREADY encodes the tolerance - an 11m
window on a 5-minute push is exactly "one missed push forgiven" - so stacking a
threshold of 3 triples a tolerance that was already chosen. Hence:
push/heartbeat endpoints failure-threshold 1
pulled, 5m (public) failure-threshold 3 (= 15 minutes)
pulled, 6h/24h (dns, domain) failure-threshold 1
── The Signal transport ────────────────────────────────────────────────────
roles/signal_api runs signal-cli-rest-api on the observability host, pinned by
digest, MODE=native.
It publishes NO PORTS. The API has no authentication of any kind - anything that
reaches it can send messages as you and read your Signal. Gatus talks to it over
a shared docker network by service name, which is also WHY the network exists:
Gatus runs in a container, so the host's loopback is unreachable from it and a
port published on 127.0.0.1 would not have worked.
MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs
Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary
per request, and alerts are rare enough that startup cost per alert is the right
trade.
Monitored - Gatus polls /v1/health over the same network path the alerts take,
so it proves the delivery route rather than mere container liveness. Deliberately
NOT backed up: the data directory holds Signal private keys and the recovery
path is to link the device again from the phone.
Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert.
If either is down, Signal is precisely what cannot deliver the alert.
── Four traps hit while linking, all now in the role README ────────────────
* /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while
the binary itself emits a perfectly good URI. Not worked around by switching
MODE, which would put a JVM in the path of every alert permanently.
* The data dir must be owned by uid 1000, not root. A root-owned 0700 dir
cannot be traversed by the container user, so linking silently never
completes and /v1/accounts returns "Failed to read local accounts list".
* `docker exec` runs as ROOT while the service runs as uid 1000, so without
--config the account is written to /root/... on the container's ephemeral
layer. It reports success and is destroyed on the next recreate.
* The phone reporting "network error" was IPv6: chat.signal.org resolves to
dualstack AAAA records first, the container has no IPv6 address, and this
host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed
with a mounted gai.conf that prefers IPv4.
Verified: provider loads (configuredProviders=[signal]), a test message was
delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
watchtower is being destroyed. Removed from [vps], with its host_vars, its push
token, and the six Gatus endpoints that referenced it (liveness, disk, two
systemd services, the ntfy DNS record and the ntfy HTTP check).
ntfy went with it - it ran nowhere else - so services/ntfy is deleted,
subdomains.ntfy and ntfy_topic are gone from group_vars, and the ntfy playbook
is out of site.yml. ntfy_topic already had no readers: the three infra/4xx plays
that used it were deleted when their checks were superseded.
Two things this exposed.
services/ntfy/deploy_ntfy_playbook.yml was pointing at the WRONG MACHINE. It
said `hosts: observability`, which resolves to the host `monitoring`
(64.226.70.190) - but ntfy ran on watchtower, and ntfy.contrapeso.xyz pointed
there. Running it would have installed ntfy on the new VPS. Moot now, but it is
the same stale-identity failure as the rest: the group meant watchtower when the
play was written, and nobody revisited it when the group changed. Watchtower was
in [vps] and NO role group at all, while running caddy, ntfy and Uptime Kuma -
nothing in the repo managed any of it.
More seriously: ntfy-emergency-app on vipy (avisame.contrapeso.xyz) sends its
notifications to https://ntfy.contrapeso.xyz, topic "emergencia". Destroying
watchtower breaks it, and it is an EMERGENCY notifier - it would fail silently
at exactly the moment it matters. That is NOT resolved here, deliberately:
standing ntfy up elsewhere, pointing at ntfy.sh, or retiring the app are all
decisions, not cleanups.
What this change does is make the break impossible to miss. The URL was derived
from subdomains.ntfy, so deleting that would have turned it into an undefined
variable buried in a template. It is now an explicit ntfy_service_url in the
app's own vars, still holding the old value, with the three options written
above it. The ntfy credentials stay in the vault because that app still needs
them - the vault was restored from HEAD and only watchtower's push token
removed, rather than re-handling the plaintext.
Verified: no reference to watchtower or its IP anywhere in the repo; Gatus down
from 91 to 85 endpoints, 85 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing in the repo pushes to, authenticates against, or is gated by Uptime
Kuma any more.
── The sixth instance of the banner bug ────────────────────────────────────
memos had `Restart memos` guarded by `uptime_kuma_enabled`, because the
deprecation banner was placed immediately above it and swept it in. It is a
HANDLER, so every memos config change since 2026-09-11 applied to disk and
silently never restarted the service. Ungated.
That is the same failure found in forgejo-runner's self-assert, phoenixd's timer
enable, mempool's three timer enables, fulcrum's restart handler and bitcoind's
restart handler. Every guard was read and asked "monitoring or deployment?"
before being deleted, which is the only reason this was caught.
── What was removed ────────────────────────────────────────────────────────
30 uptime_kuma_enabled guards across 7 unconverted service playbooks, and
the 29 Kuma monitor-creation tasks they gated (embedded Python that drove
the Kuma API, temp credential files, cleanup)
7 dead uptime_kuma_api_url definitions
7 stale DEPRECATED banners
uptime_kuma_enabled and subdomains.uptime_kuma from group_vars/all
healthcheck_push_urls from the vault - 30 push tokens
services/ntfy/setup_ntfy_uptime_kuma_notification.yml -> archive/
The explanatory comments in the six converted roles are KEPT on purpose. They
record why a handler is ungated, and deleting the explanation invites someone
to helpfully re-add the guard.
── The probes moved rather than died ───────────────────────────────────────
Eight per-service health checks were still pushing to Kuma. They are not
superseded by infra/401: that answers "is the unit running", these answer "does
the service actually respond" - an RPC call to bitcoind, a TCP connect to
Fulcrum's Electrum port, an HTTP fetch from Mempool's backend. A process can be
perfectly `active` and useless.
So they were repointed, not deleted. Gatus external endpoints take a POST with
a bearer token and success=true|false where Kuma took a GET with ?status=up, so
report() now maps up/down to true/false internally and no call site changed.
Registered by infra/403 as the `probe` group, one token per host.
Two bugs fixed while in there:
* forgejo-runner's check only ever reported SUCCESS - it exited before pushing
when the runner was down, so a failure was invisible until the heartbeat
window expired. Reporting the failure is the entire point of a check.
* All six healthcheck .service units were mode 0644 and now carry a bearer
token. They are 0600.
Verified: 91 endpoints, 91 UP, 0 DOWN. Every probe triggered by hand and
confirmed arriving. Zero Kuma URLs left in the vault, zero live references in
any playbook or role.
Still standing, deliberately: the Kuma container on watchtower, its Caddy vhost,
and the uptime.contrapeso.xyz DNS record. Turning the service off is a separate
decision from removing the code that talked to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five checks, 27 endpoints, replacing what Uptime Kuma used to watch:
is it up every 5min, all hosts
is disk full daily, all hosts
is CPU hot every 5min, nodito
is ZFS broken daily, nodito
is UPS online every 5min, nodito
Two roles, kept separate so neither knows about the other - they meet at a URL
and a token, the same way caddy_site and each service meet at a vhost:
roles/gatus_endpoint runs on the observability host, writes ONE file into
/opt/gatus/config/endpoints/. Gatus merges every *.yaml
there and appends lists, so callers compose without
coordinating.
roles/healthcheck runs on the monitored host: a check script, a systemd
service, a timer, and an optional push. Ships a library
of check bodies under templates/checks/.
Everything PUSHES. Gatus never reaches out, which matters because nodito and its
VMs are behind NAT, and because four of the five checks are internal state with
no pollable surface at all. Liveness pushes too, deliberately: a heartbeat proves
the host is running AND can reach the internet, where an ICMP probe from one
vantage point only proves it answers pings from there. And since Gatus alerts
when a heartbeat window expires, a check that stops running raises the alarm by
itself - a dead timer looks exactly like a dead host, which is the correct
reading.
One bearer token per host, generated straight into the vault and never printed.
A token only writes results for its own host's endpoints, so a compromised host
can lie about itself, which it could do anyway.
Three things learned from the source that shaped this:
* Gatus polls its own config every 30s and reloads
(main.listenToConfigurationFileChanges), so gatus_endpoint needs no restart
handler - writing the file IS the deploy.
* ...but on a reload it panics if the new config fails to parse, unless
skip-invalid-config-update is set. Endpoint files are contributed by other
playbooks, so one malformed file would take the monitor down at the worst
possible moment. Now set.
* The push URL uses a key Gatus computes, not the name you write:
sanitize(group) + "_" + sanitize(name), lowercased with / _ . , space # + &
replaced by "-" (config/key/key.go). So knots_box_local is
knots-box-local in the URL. The playbook derives it rather than hand-writing.
storage: maximum-number-of-results 900, up from upstream's 100. Gatus bounds the
database by COUNT and trims inline on insert, so there is no retention job and
no way to fill a disk - but history depth is then a function of check frequency,
and 100 results at a 5-minute interval is 8 hours. 900 is ~3 days of liveness
and ~2.5 years of the daily disk check. The uptime table is separate and its
30-day retention is hard-coded upstream.
A bug worth recording: the first deploy shipped five scripts that all died with
"syntax error: unexpected end of file". Jinja strips an included template's
trailing newline and trim_blocks then eats the newline after {% endif %}, so the
closing brace of check() landed on the same line as the body's last statement -
`return 0}`. Every check was broken and the deploy still reported failed=0,
because the role's "run once" task has failed_when: false and reports the result
as a debug message nobody read. The blank line that fixes it is now load-bearing
and commented as such.
Verified by triggering every unit by hand rather than waiting on timers: all
checks exit 0 on all hosts, and Gatus shows 26 UP / 1 DOWN. The one DOWN is
liveness_watchtower, which is honest - that host currently refuses SSH (TCP
connects, no banner exchange) and is excluded from this deploy. It is also the
box still running Uptime Kuma.
Known waste, not yet fixed: healthcheck installs its dependencies per CHECK
rather than per HOST, so apt runs 29 times estate-wide for a curl that is
already present, and daemon_reload runs 4x per host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First step of replacing Uptime Kuma and ntfy. Gatus runs on the new
observability VPS, fronted by Caddy at status.contrapeso.xyz.
Deployed as the upstream container image, not built from source. The role did
build from source first - their Dockerfile is a bare `CGO_ENABLED=0 go build`,
the Vue dashboard is compiled in via `//go:embed static` in web/static.go, and
CGO can stay off because the sqlite driver is pure-Go modernc.org/sqlite - but
that produces a binary upstream never ran, and it meant compiling the AWS SDK
and gRPC on the smallest box in the estate. That load was heavy enough that
unrelated Ansible tasks timed out while it ran. The cost of the container is a
daemon on the machine whose job is to notice when everything else breaks; that
trade is made deliberately and is written down in the role README.
Pinned by DIGEST, not tag. A tag is mutable - v5.36.0 can be repushed - so
pinning it alone is a weaker promise than it looks:
gatus_image: "ghcr.io/twin/gatus@sha256:c5f210d0..."
`docker compose pull` now either fetches exactly the reviewed image or fails.
gatus_version is kept beside it only so a human can read the release; the two
move together.
The image is FROM scratch, so it has no /etc/passwd and its default user is
root. The container runs as 10001:10001 with the host data dir owned to match,
plus read_only, cap_drop ALL, and no-new-privileges. NET_RAW is added back only
when gatus_allow_icmp, so the capability for icmp:// checks is a visible grant
rather than something inherited from running as root.
Config is a DIRECTORY, not a file. Gatus merges every *.yaml under
GATUS_CONFIG_PATH - maps deep-merge, lists append - so the role owns
00-base.yaml (web, storage, ui, alerting, security) and each service will drop
its own file into endpoints/, the same shape as caddy_site. A primitive defined
twice is ambiguous and upstream refuses it, so anything that is not a list lives
in the base file and nowhere else.
Two bugs the deploy caught:
* Gatus panics on a config with no endpoints ("configuration should contain at
least one endpoint or suite"), so "install now, add endpoints later" is not
a valid state. The role ships endpoints/00-self.yaml checking its own
/health. Less circular than it looks: it proves the directory merged, the
listener serves, and storage accepted a write.
* web.address was carried over from the systemd design as 127.0.0.1. Inside a
container that is the CONTAINER's loopback, which docker-proxy cannot reach
- gatus came up healthy, self-check passing, while every connection to the
published port was refused. It now always binds 0.0.0.0 inside the
container; the isolation comes from publishing to 127.0.0.1 on the host.
Auth is done at the edge, NOT with Gatus's own security.basic. Reading
api/api.go, that middleware protects exactly four routes - the statuses
endpoints. Everything else is registered on the unprotected router, including
/api/v1/config, every badge, and /api/v1/endpoints/:key/uptimes/:duration and
.../response-times/:duration/history, which return real data to anyone who can
guess a key ("<group>_<name>"). Verified against the live instance: all seven
routes returned 200 unauthenticated, and /uptimes/24h returned "1.000000".
So the vhost uses caddy_site_body with a path carve-out rather than
caddy_site_basic_auth, which has no way to exempt a path. The external-endpoint
push API must NOT sit behind basic auth: it authenticates with
`Authorization: Bearer <token>`, and basic auth wants the same header. It is not
unauthenticated - the handler 401s on a missing prefix, an empty token, or a
token that does not match that endpoint's own.
Verified end to end. All seven previously-open routes now 401. The push path
distinguishes cleanly: POST with no auth gets Gatus's own "invalid Authorization
header" with NO WWW-Authenticate; POST with a bogus Bearer gets 404 (key looked
up, no external endpoints yet); GET on the same path gets Caddy's 401 with
WWW-Authenticate: Basic, so the exemption is scoped to POST alone. The
self-check still passes because it polls localhost inside the container and
never traverses Caddy.
The host itself was rebuilt from scratch: 01 (ok=9 changed=8), 02 (ok=12
changed=6), 910_docker (--limit, since that playbook still wrongly claims all of
`managed` needs Docker), caddy (ok=13 changed=8), gatus (ok=18 changed=2).
Not done here: gatus_alerting is still {} - valid, and every condition is
evaluated and recorded, there is just nowhere to shout until a provider is
chosen to replace ntfy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These two are host-specific by design - nodito is a pet, not cattle - so they
stay playbooks rather than becoming roles. But they had rotted.
De-Kuma, following the pattern of the six service roles:
Both plays opened with an assert on uptime_kuma_username/password, which were
removed from the vault, so both failed before doing anything. Dropped that,
the two embedded Python monitor-creation scripts, and their /tmp cleanup.
Kept every check, threshold and systemd timer - those are the durable part.
Reporting is now generic: `healthcheck_push_url` goes into the unit as
Environment=HEALTHCHECK_PUSH_URL and the script reads ${HEALTHCHECK_PUSH_URL:-},
treating empty as normal rather than an error. The exit code is the real
answer; systemd keeps it. Both scripts now also report status=down on failure
instead of only going silent. Live push URLs harvested into the vault so
nothing observable changes for ZFS.
Three live bugs found while check-diffing:
1. 32_zfs would have DE-REGISTERED the Proxmox storage. `pvesm remove` was
gated on the storage existing and `pvesm add` on it NOT existing - mutually
exclusive - so a real run removed the proxmox-tank-1 entry backing every VM
and never put it back. It would also have dropped `mountpoint /var/lib/vz`,
which the live entry has and `pvesm add` does not set. Registration is now
add-only.
2. zfs_disk_1 named ata-...WX11TN0Z, a disk no longer in the machine. The live
mirror is WX120LHQ + WX11TN2P; a leg was replaced and the repo never caught
up. Inert behind `when: zfs_pool_exists.rc != 0`, but wrong on any
disaster-recovery run. This is the seventh instance of an identifier
written down once whose hardware later moved.
3. 34_nut has NEVER been applied to nodito - no /etc/nut file carries the
"Managed by Ansible" marker; they were written by hand in January 2026. The
vault held the literal CHANGE_ME_TO_SECURE_PASSWORD, so applying it would
have overwritten a working upsd/upsmon auth pair with a placeholder and
restarted NUT, leaving the hypervisor's UPS unable to trigger a clean
shutdown on mains loss. The Kuma assert was the only thing stopping that,
so removing it without a replacement would have armed the gun: there is now
an explicit assert that refuses to run on the placeholder. The real
password is in the vault and `Configure upsd users` check-diffs clean.
Templates reconciled with the live files first, so applying 34_nut is close to
a no-op: added `maxretry = 3` to ups.conf and OFFDURATION / RBWARNTIME /
NOCOMMWARNTIME / FINALDELAY plus quoted POWERDOWNFLAG to upsmon.conf. The only
substantive additions left are the NOTIFYMSG/NOTIFYFLAG syslog lines.
/usr/local/bin/ups-heartbeat.sh on the box is an orphan - mode 0644, not
executable, referenced by no unit and no cron entry - but its push token
belongs to a monitor that still exists and answers, so that monitor has had no
heartbeat since January. Harvested as healthcheck_push_urls.ups; applying this
play is what will finally feed it.
Finish the host_vars migration:
infra/nodito/nodito_vars.yml was byte-identical to host_vars/nodito/main.yml.
Deleted it, moved nodito_secrets.yml to host_vars/nodito/vault.yml, and
stripped the dead vars_files entries from all three playbooks.
Extract the 13 inline `content: |` blocks to infra/nodito/templates/, pulled via
a YAML load rather than retyped. 681+582 lines become 313+344 plus templates.
Ownership parity against HEAD checked mechanically: no owner/group/mode drift on
any surviving task.
Neither playbook has been applied. ZFS play 2 check-runs failed=0 with two
changes: the script rewrite (the deployed one is a hand-edited DEBUG VERSION
with `set -x`) and the Environment line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
802-line playbook becomes 68 lines (three plays: the role, the Caddy dashboard,
the Stratum socket proxy) plus a 345-line role. datum_gateway_vars.yml is
deleted; its content is the role's defaults.
Verified after a real run with zero miners connected: datum-gateway restarted
cleanly onto the reformatted config, deployed config.json semantically identical
to what was there (pool_address bc1qvrj3g84..., pool_pass_* false, ports
unchanged), health check timer firing, and the Knots side untouched - bitcoind
still up since 2026-08-19 with blocknotify intact.
TWO PIECES OF DRIFT WHERE THE NODE WAS RIGHT, both confirmed with the operator:
- datum_mining_address: the vault held bc1qdse9dsg... while the node had been
mining to bc1qvrj3g... since 2026-08-08. This is WHERE BLOCK REWARDS ARE PAID.
And unlike fulcrum and bitcoin-knots, the `Restart datum-gateway` handler here
was never gated, so the stale value would have applied immediately rather than
sitting inert on disk.
- pool_pass_workers / pool_pass_full_users: false on the node, true in the vars
file.
Both corrected in the vault and role defaults with notes recording why.
Comparing this config needs semantics, not text: the live file is single-line
JSON and the template renders pretty-printed, so a textual diff is pure noise.
Rendering it and comparing parsed JSON is what surfaced both differences.
config.json carries bitcoind.rpcpassword and api.admin_password, and --diff
prints rendered content - so `--check --diff` put them on the terminal. The task
now sets diff: false by default (-e datum_reveal_config=true to opt in). Those
two should be rotated.
I also mis-reported pool_pass_workers/pool_pass_full_users as exposed credentials
because my masking matched "pass" in the key name. They are BOOLEANS, and
mining.pool_address is a Bitcoin address, public by nature. Only the two real
passwords above were exposed.
`Configure cmake build` and `Compile datum_gateway` are bare command: tasks with
no changed_when, so they recompile on every run. The build is reproducible -
Install datum_gateway binary sees identical content and leaves the installed
binary's timestamp alone - but it is wasted work each time. Documented as the
idempotent floor.
Ownership parity checked mechanically against `git show HEAD:` keyed by task
name: 7/7 match, 9 Kuma tasks dropped.
This completes Plan 6 Stage 2: all six services in the list are roles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
892-line playbook becomes 40 lines plus a role with install/build/configure/
service/healthcheck phases, five templates and one handler. bitcoin_knots_vars.yml
is deleted; its content is the role's defaults.
Verified after a real run: bitcoind still active since 2026-08-19 (NO restart),
chain at 966844 blocks / 875 GB, DATUM config intact, dbcache still 200, health
check timer firing again. changed=3, all health-check. vipy changed=0.
⚠ THE BIG ONE: the playbook would have deleted the mining integration.
bitcoin.conf on the node carries a section that was hand-added and was missing
from the template entirely:
blockmaxsize=3985000
blockmaxweight=3985000
blocknotify=killall -USR1 datum_gateway
maxmempool=1000
blockreconstructionextratxn=1000000
blocknotify is how datum_gateway learns a new block landed. Running the old
playbook would have stripped all of it and solo mining would have carried on
against a stale template - a silent failure that costs money rather than raising
an error. Also dbcache 200 -> 3528 (hand-tuned down; the calculation wants 90% of
RAM) and logging moved off the file. All now reconciled, dbcache behind
bitcoin_dbcache_mb_override.
bitcoin-knots and datum-gateway are ONE SYSTEM. Noted in the README.
AND MY OWN FIX MADE IT MORE DANGEROUS. The `Restart bitcoind` handler was guarded
by uptime_kuma_enabled, so it had been inert: bitcoin.conf and the systemd unit
both notify it and neither could restart anything - a config change applied to
disk, reported success, and never took effect. Ungating that is right, but it
converts "wrong config sitting inertly on disk" into "node restarted onto a
config that breaks mining". The ungating had to land WITH the template
reconciliation, not before it.
It also raises the bar permanently: any residual template/live difference now
restarts a Bitcoin node on every run. Four rounds of --check --diff to reach
changed=0 - the DATUM section, an explanatory comment that was rendering into the
deployed config (now a {# #} Jinja comment), a "# Pruning (optional)" comment the
live file had, and one trailing blank line.
The build path is 32 tasks all guarded by `not bitcoind_binary_exists.stat.exists`,
so a converged host skips the 30-60 minute compile and both `state: absent`
deletions. Those target /opt/bitcoin-knots/{source,bitcoin-<version>}; the chain
is in /mnt/knots_data and is never touched. Signature-verification tasks copied
verbatim.
The health check timer had last fired 2026-08-09 while reporting active/enabled -
same OnBootSec + OnUnitActiveSec dead chain as fulcrum. The role runs the check
once after enabling to supply the reference the timer schedules from.
Ownership parity checked mechanically against `git show HEAD:`, keyed by TASK
NAME rather than path - keying by path gave a false positive, because
bitcoin_knots_source_dir is created with ownership and later removed with
state: absent, so whichever task comes last wins and that differs between one
file and five. 13/13 match.
bitcoin_p2p_port and the tailscale hostname moved to services_config.yml for the
socket-proxy play on the edge host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
745-line playbook becomes 37 lines (the role, plus the Caddy play for the edge
host) and a 408-line role with docker/deploy/healthcheck phases and six
templates. mempool_vars.yml is deleted; its content is the role's defaults.
Three health checks are kept, not collapsed: Mempool is three moving parts and
knowing which one is down is the point. Each has its own script, unit, timer and
push_url, driven by a mempool_healthchecks list. The Uptime Kuma specifics are
gone - the embedded Python creating monitors over the API, the /tmp credentials
file, the push-URL file read back and parsed, three Environment= rewrites - and
the three live push URLs are preserved from the vault, so reporting is unchanged.
`Enable and start health check timers` and `Display deployment status` were both
guarded by uptime_kuma_enabled despite being deployment tasks. Third service in
a row with that pattern: the deprecation banner was applied to contiguous blocks,
so anything sitting near the push plumbing was disabled with it. Ungated.
TWO OWNERSHIP PROBLEMS, different in kind:
- MINE: I wrote `owner: root` on docker-compose.yml where the original says
`owner: "{{ ansible_user }}"`. A straight violation of extract-mechanically-
change-nothing, caught only by reading the check-mode diff line by line.
Reverted to match the original.
- PRE-EXISTING, and dangerous: the playbook declared
`owner: "{{ ansible_user }}"` (1000) on the MariaDB data directory, which the
container owns as uid 999. Confirmed against `git show HEAD:` before
concluding it was not mine. It had drifted since the containers were created
and went unnoticed because the playbook had not been run since.
This was not academic. The first real run pulled a newer mariadb:10.11 and
recreated mempool-db; with the chown still in place MariaDB would have come
back to a data directory it could not write. The role now ensures the
directory exists and leaves ownership to the container. Verified after the
run: /opt/mempool/mysql is still 999:999 and all three containers are healthy.
This is a deliberate behaviour change, not part of the extraction. It is in
this commit rather than a follow-up because the faithful version was never
safe to run, so there was no intermediate state worth recording as verified.
mempool_frontend_port moved to services_config.yml: two hosts need it (this role
deploys the frontend, the Caddy play proxies to it from the edge host) and a role
default is invisible to the second play. caddy_site's parameter assert caught
this loudly - "'mempool_frontend_port' is undefined" - rather than silently.
Verified: check-mode diff clean apart from unavoidable check-mode artifacts;
first run ok=24 changed=5, zero failures; second run changed=2 - the two bare
`command:` tasks (pull, compose up) that have no changed_when and always report
changed. That is the idempotent floor. All three health checks report
ExecMainStatus 0 with their push URLs intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
409-line playbook becomes a 16-line playbook plus a 318-line role with phases
split across tasks/{prerequisites,install,configure,service,healthcheck}.yml and
four templates. forgejo_runner_vars.yml is deleted; its content is the role's
defaults.
Applies the Plan 6 Stage 0 decision: keep whatever determines whether the
service is healthy, drop the Uptime Kuma specifics, make the reporting point
pluggable. Gone from the role: the embedded Python that created monitors over
the Kuma API, the /tmp credentials file, token extraction, the systemd
Environment= rewrite, and 8 `when: uptime_kuma_enabled` guards. What remains is
the check itself, its log, the systemd unit and timer, and an honest exit code -
`systemctl is-failed forgejo-runner-healthcheck.service` now answers the
question with no monitoring system involved at all.
Reporting is one variable, healthcheck_push_url, empty by default. Any endpoint
that accepts an HTTP ping plugs in there. A pull-based monitor wants it left
empty and reads unit state instead.
PREMISE CORRECTION: Uptime Kuma is NOT dead. Plan 3 recorded "48 push timers
curling an endpoint that no longer answers" and Plan 6 said the check had
"nowhere to report to". Both wrong - 24+ push scripts across 11 hosts are
pushing successfully right now (HTTP 200). Only the Ansible code and the vault
credentials were decommissioned; the service never stopped. So the existing push
URLs were harvested into a vaulted healthcheck_push_urls dict and are preserved,
keeping this refactor behaviour-neutral. Retiring Kuma stays a deliberate act
rather than a side effect. PLAN_3 and PLAN_6 are corrected.
Verified:
- task-list diff vs the old playbook shows ONLY the five Kuma tasks removed,
everything else identical and in the same order
- first run ok=22 changed=1 (the rewritten health script); both systemd units
and forgejo-runner.service came back ok, so the templates reproduce the
previous files byte-for-byte
- second run ok=22 changed=0, fully idempotent
- still reports "Ping sent successfully (HTTP 200)" from a script containing
zero Uptime Kuma references
- the 4 skipped tasks are genuine already-configured guards, checked not assumed
Two things for the next service:
- import_tasks, not include_tasks. Dynamic includes are opaque to --list-tasks,
which is the primary verification tool here; the first attempt produced a
useless diff.
- `Assert runner is running` was guarded by uptime_kuma_enabled and so had not
run since the decommissioning. It is not monitoring, it is the deployment
checking its own work - the deprecation banner swept it up with the Kuma
plumbing, and a runner that failed to start was deploying "successfully" in
silence. Ungated now. The banner was applied to contiguous blocks, so read
every uptime_kuma_enabled guard and ask whether it is monitoring or deployment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes Stage 3. No hand-rolled Caddy plumbing remains anywhere:
`grep sites-enabled` outside roles/ returns nothing, and so does
`grep "systemctl reload caddy"`.
ntfy uses caddy_site_body for its plain-HTTP listener and @httpget redirect.
Verified ok/unchanged against watchtower; the one other changed task is a
pre-existing "Update APT cache".
datum-gateway keeps a whole-Caddyfile validate after the role call. The role
validates its own fragment, but only a whole-file validate catches a conflict
between two sites, and this playbook was the only one that ever had it. Its
two debug tasks that echoed command output are gone with the commands.
headscale is the one that mattered. Its playbook wrote
`reverse_proxy localhost:8080`, but spacey is actually running a /admin*
route in front of Headplane behind Caddy basic auth. Running that playbook
would have deleted the admin route and its auth - a hazard that predates this
work. It now renders the config that is really there, verified ok/unchanged
via --start-at-task (the play cannot reach Caddy in check mode: "Install
headscale package" fails because the .deb is not really downloaded, before
and after this edit alike).
Supporting changes for headscale:
- headscale_ui_password_hash added to infra_secrets.yml and the identical
group_vars/all/vault.yml, read from the live config on spacey. The vault
already had headscale_ui_username (= counterweight, confirmed) and
headscale_ui_password; I did not verify the password is the plaintext of
this hash.
- headplane_port added to headscale_vars.yml.
- The role's handler now sets become: true. Handlers do not inherit become
from the task that notified them, and this play runs become: no.
- The include uses `apply: become: yes`; `become:` on an include_role is
rejected outright.
All 14 site files on all 3 hosts still byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops uptime_kuma_username/password from infra_secrets.yml and its group_vars
copy and example, plus a dead push token in nodito_secrets.yml that nothing
referenced. They remain in git history — rotation is what actually retires them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
