All four droplets were deleted by mistake on 2026-10-17 and rebuilt as two:
Gatus and signal-api now share spacey with headscale, and arbret moves onto
vipy. The old IPs were recycled to strangers within a day.
- inventory: [observability] is spacey. Left as it was, the bare name
`monitoring` resolved through ~/.ssh/config to an IP that now belongs to
someone else. [arbret]/prd-arbret is gone.
- inventory: lan_ip fallbacks corrected from the Proxmox guest agent -
memos-box is .130 (was .145) and arbret-staging-box .145 (was .147).
- spacey: gatus_port 8081. headscale already holds 0.0.0.0:8080, and the two
never shared a host before.
- 402: status.<domain> is expected to resolve to spacey.
- headscale ACL: prd-arbret, prd-monitoring and stalwart-box removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.
── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.
── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.
That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.
── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.
── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.
The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.
Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.
── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.
── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
/health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.
── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.
Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.
The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.
A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.
── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.
── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.
/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.
── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).
/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.
── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".
inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.
50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.
No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A wedged explorer produced 83 Signal messages in six hours. Nothing about that
was a monitoring success: the checks were right, the cadence was wrong.
monerod-box is a Monero node in the middle of a multi-week initial sync on a
rotating disk. A 5-minute poll tells you nothing a daily one does not, and when
something breaks it turns one fault into a message every five minutes until
somebody silences the phone - which is the failure mode that makes people stop
reading alerts entirely.
So both Monero probes move to OnCalendar:
monerod *-*-* 05:00:00 RandomizedDelaySec=900
xmrblocks *-*-* 05:30:00 RandomizedDelaySec=900
systemd units (infra/401 per-host timer) *-*-* 05:15:00
30 minutes apart on purpose: both read the same LMDB, and the explorer's check
renders a full page out of it - a cold one measured 68 seconds. Running them
together would have each paying the other's cache misses.
OnCalendar rather than OnUnitActiveSec: an interval timer re-bases on every
restart, so "daily" silently becomes "a day after whenever it last came up".
── The window and the cadence are ONE decision ─────────────────────────────
The Gatus heartbeat moves with them, to 26h. Not 24h: the window must EXCEED
the push interval, or a check reporting exactly on schedule races its own
deadline and alerts on jitter alone. With a 900s randomized delay the true
worst case is ~24h15m, so 26h leaves real headroom.
The same logic drops the pulled public-site checks to a 24h interval - and
their failure-threshold from 3 to 1 with it. Three failures at 5m is 15 minutes
of tolerance; three at 24h would be THREE DAYS before anyone heard. A daily
check has no room for a blip, so it alerts on the first.
── Scoping ────────────────────────────────────────────────────────────────
infra/401's timer is ONE PER HOST and iterates every unit in
monitored_services, so slowing it slows all of them. That is safe here only
because monerod-box runs nothing but Monero units; the knobs are therefore in
its host_vars rather than the role, and both 401 and 403 now take a per-host
(resp. per-probe) override with the old 11m as the default. Every other host
is untouched.
Host liveness and disk are deliberately NOT slowed. They answer "is the box
alive", which is still wanted within minutes.
The role-level OnCalendar defaults are prefixed (monero_healthcheck_on_calendar,
xmrblocks_healthcheck_on_calendar) because the generic `healthcheck` role owns
the unprefixed name, and an unprefixed default here would shadow it for any
play that includes both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Onion Monero Blockchain Explorer on monerod-box, published at
exmrplore.contrapeso.xyz. Ten Gatus endpoints green across the node and the
explorer.
── It needs a compiled monero source tree, not a running monerod ───────────
xmrblocks is not an RPC client. cmake/FindMonero.cmake imports 21 STATIC
libraries out of a monero build tree and includes its headers, because it opens
the LMDB directly. The release tarball we install for the daemon ships only
executables - no .a files, no headers - so there is nothing on the box to link
against, and building it means compiling monero first.
Measured on Debian 13, building only the libraries it links:
peak RSS, one cc1plus ~3.2 GB
peak RSS at -j4 ~5.5 GB
wall time at -j4 167 s
source + build tree 195 MB
toolchain 1.6 GB
That rules the box out at ANY parallelism: 768 MiB and 1800 MiB were both
OOM-killed at -j1 and -j2, because a single translation unit over 3 GB is not
something lower -j works around. My first estimate of "~10 GB of disk" was
wrong by two orders of magnitude - that figure describes upstream's full `make`,
which also builds monerod, both wallets and the test suite, none of it linked
here. Disk was never the constraint.
So scripts/build-xmrblocks.sh does the build in a debian:13 container. An
artefact only one machine can produce is one nobody can audit; this way any host
with Docker and ~6 GB reproduces it. It verifies the monero source against
binaryFate's signature exactly as roles/monero does for the daemon, and the role
asserts the resulting sha256 ON THE CONTROL NODE before copying a single byte.
── Three upstream problems, none documented anywhere ───────────────────────
Found by building, not by reading:
1. libasio-dev is absent from upstream's dependency list. That list is
Ubuntu-based; Crow wants standalone asio.hpp, not boost/asio.hpp, and on
Debian 13 the build dies outright.
2. The explorer does not compile against monero v0.18.5.1. v0.18.5.0 changed
get_pruned_transaction_hash from returning the hash to returning bool with
an out-param, and src/page.h still calls the old form. Upstream's
Dockerfile pins v0.18.4.0, which is why nobody has hit it. Patched from an
exact-line anchor, so if upstream fixes it the build STOPS rather than
quietly doing something else. We patch instead of building against 0.18.4.0
because the signed hashes.txt only covers the current release - using the
older source would mean dropping the provenance chain. LMDB is VERSION 5 in
both, so either reads the other's database.
3. FindMonero.cmake hunts for miniupnpc inside the monero build tree with
NO_DEFAULT_PATH, but monero stopped vendoring miniupnp as a submodule.
Fixed with a cache override rather than patching their cmake.
USE_DEVICE_TREZOR=OFF as well - an explorer has no use for hardware wallets and
it drops the generated protobuf units. Verified that xmrblocks still links.
── Two things in the unit that look wrong and are not ──────────────────────
WorkingDirectory is load-bearing: page.h hardcodes TMPL_DIR as "./templates",
resolved relative to cwd at REQUEST time. Get it wrong and the service starts,
binds 8081, and 500s every page. That is also why the health check fetches a
real page and greps it instead of probing the port.
ReadWritePaths must include the blockchain directory even though the database is
opened read-only: monero sets mdb_flags = MDB_RDONLY WITHOUT MDB_NOLOCK, so
LMDB still registers this reader in lock.mdb. With ProtectSystem=strict and
nothing writable the open fails. "It only reads, so grant it nothing" is wrong.
── Timeouts are measured, not guessed ──────────────────────────────────────
proxmox-tank-1 is a spinning-disk mirror and xmrblocks says so on startup.
First start took 98s to open the LMDB before binding (wait_for now allows 300s),
and a COLD index page took 68 SECONDS where the next two took 0.87s and 0.012s
(the check now allows 90s, its unit 180s). The first pass used 90s and 25s and
produced exactly the false failures you would expect.
A stat() bug was caught the same way: `stat` computes a sha1 by default, so
checking that data.mdb exists was hashing the entire 40 GB blockchain. --check
found it before a real run did.
── Exposure ────────────────────────────────────────────────────────────────
Caddy on vipy terminates TLS; the upstream is reached over the tailnet, which
needed a headscale ACL entry for tag:monerod-box:8081 - without it the node is
invisible, not merely unreachable, and the symptom is a 502 "no such host".
No authentication, deliberately: it is a block explorer over public chain data.
That includes --enable-pusher, so a visitor can relay a raw transaction through
our node. A choice, not an oversight; flip xmrblocks_enable_pusher to close it.
── Also ────────────────────────────────────────────────────────────────────
monerod-box raised 2048 -> 6144 MiB. Not for the explorer: monerod alone had
outgrown 2 GB past ~39% of the chain and was OOM-killed twice in a morning at
1.74 GB anon-rss. Recorded in vms.auto.tfvars because ignore_changes = all means
tofu will not reconcile it. The box now idles at ~1.2 GB of 5.9 GB with both
services up.
No backups, deliberately - binary, templates and chain are all reproducible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
monerod v0.18.5.1 on monerod-box (VMID 2400), Debian 13, systemd, no Docker.
Syncing at 24% as of this commit, 0 restarts, 0 OOM kills.
── The build-from-source requirement could not be met ──────────────────────
The intent was to build it ourselves and check the result against the published
checksums. That is not achievable for this release, and the role's README says
so at length so nobody re-derives it:
contrib/guix - the reproducible build that yields a byte-identical tarball -
exists ONLY on monero master. At tag v0.18.5.1 contrib/ contains `gitian`,
whose README wants an Ubuntu 18.04 host (EOL 2023) and python-cheetah.
A plain `make release-static` is not reproducible at all, so it produces a
binary whose hash matches nothing - failing both halves of the ask.
So we install the published binary and prove its provenance instead:
1. binaryFate's key is VENDORED at files/binaryfate.asc. A trust root fetched
over the network at the moment you need it is not a trust root. Its
fingerprint is asserted after import regardless.
2. hashes.txt is verified, and the assertion is on gpg's `VALIDSIG <fpr>`
status line - NOT on the string "Good signature", which is what
bitcoin_knots greps for and which passes for any key in the keyring.
3. The hash is looked up in the output of `gpg --decrypt`, not in the raw
clearsigned file, so nothing unsigned is ever searched.
4. The lookup requires hash AND filename on one line.
Tested with a positive and a negative control before deploying: the real
tarball matches, a corrupted hash correctly fails. monero_sha256 in defaults/
is a second in-git assertion so a version bump cannot happen without the
checksum moving in the same diff.
When v0.19 ships with guix on the release branch this gets easier, and the
README records why: the published monero-linux-x64 tarball unpacks to
monero-x86_64-linux-gnu-v0.18.5.1/, i.e. it IS the guix DISTNAME renamed, so
sha256(guix output) will equal the hashes.txt entry exactly.
── The 500 GB disk is provisioned in code ──────────────────────────────────
knots-box mounts /mnt/knots_data (~875 GB of chain) entirely by hand; that
mount appears nowhere in Ansible and a rebuild of that host would come up with
the blockchain silently missing. tasks/storage.yml does not repeat it.
Whole device, no partition table. Everything after mkfs goes by UUID, because
/dev/sdX is assigned by the kernel at boot and this estate has been bitten
eight times by an identifier written down once whose thing then moved.
monero_data_device is used ONLY to create the filesystem. Guards refuse to
touch the root disk or to overwrite an unrecognised filesystem, and a final
assert confirms the data directory is not still on /.
fstab gets `nofail` so a missing disk cannot drop a headless VM into an
emergency shell - and the unit gets `RequiresMountsFor=`, without which monerod
would then start, find an empty directory, and write a 300 GB chain onto a
10 GB root disk. With it the unit refuses to start and infra/401 says so within
five minutes.
The mount-point task and the ownership task deliberately share owner and mode.
Writing the obvious root:root 0755 on the first made them disagree and every
run flipped the directory back and forth; caught by re-running --check.
── Exposure: one HTTPS vhost, zero new open ports ──────────────────────────
Two RPC servers, not one: unrestricted on 127.0.0.1:18081 for administration
and the health check, restricted on :18089 for everything else. That is
`rpc-restricted-bind-port` and deliberately NOT `restricted-rpc=1`, which would
restrict the loopback server too. Verified live: the restricted port reports
restricted=true and blanks version and start_time.
--rpc-login is HTTP Digest (contrib/epee/src/http_auth.cpp) and
core_rpc_server::init hands the same login to both servers. Caddy on vipy does
TLS and routing only - no caddy_site_basic_auth, because a Basic challenge
would collide with the Digest challenge Feather answers. Verified end to end
from the WAN: 401 unauthenticated, get_block_count over /json_rpc when
authenticated, 401 again with a wrong password.
No UFW rule, and that is correct: tailnet traffic bypasses UFW entirely because
`-A INPUT -j ts-input` sits ahead of every ufw-* chain. Confirmed on mempool-box
before relying on it here.
P2P is outbound only. Nothing forwards 18080, so the node dials out, syncs and
serves us without relaying for strangers; no-igd follows from that.
── Monitoring: infra, systemd, public. No backups ──────────────────────────
Six Gatus endpoints, all green: liveness, disk, services/monerod-box-local/
monerod, probe/monerod, dns/monero.contrapeso.xyz, public/monero.
The probe does NOT assert `synchronized`. A full sync takes days and a check
requiring it would report DOWN for every one of them - an alarm with the
batteries taken out. Height and sync state go in the message instead.
The timer carries no `Requires=monerod.service`. Fulcrum's did, and sat
active+enabled for seven months without firing once. public/monero asserts 401,
not 200, so it goes red if rpc-login ever stops being enforced.
No backups, deliberately, following the phoenixd precedent: the chain is
re-syncable public data and there is no wallet on this node.
── Known constraint ────────────────────────────────────────────────────────
The VM has 2 GB and no swap; monerod's RSS is ~1.3 GB mid-sync. It is holding
(0 OOM kills, ~1 GB available, kernel reclaiming cache) but 8 GB is the
recommendation once the initial sync is done. out-peers/in-peers are held at
16/8 partly for this reason. Bumping it is a Proxmox resize plus memory_mb
here, since the vm resource sets ignore_changes = all.
The 500 GB data disk is likewise invisible to tofu - it was attached in the UI
and the vm resource ignores it - so vms.auto.tfvars now carries a comment
saying where it is and what mounts it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes: detection windows tightened, a Signal transport deployed, and
alerts attached to all 84 non-transport endpoints.
── Failing faster ──────────────────────────────────────────────────────────
The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks
"did anything arrive in the last interval", so real detection is 1-2x the
window. And a window can only ever be as tight as the push frequency - which is
why two checks moved rather than just having their numbers changed.
liveness, cpu, ups, service-health, probes 16m -> 11m
disk, zfs daily/30h -> 6-hourly/7h
backup store + pull job daily/30h -> 6-hourly/7h
DNS records 24h -> 6h
backup dump 30h -> 26h
backup-dump stays slow because the dump genuinely is daily. The store-side check
catches the same fault within 6h by reading the source's dump timestamp out of
the artefact filename, so 26h is a backstop rather than the primary signal.
── Thresholds differ by check type, deliberately ───────────────────────────
`failure-threshold` counts consecutive failures, but "consecutive" is a
different amount of wall-clock time per check: a push endpoint produces one
failure per heartbeat window, a pulled one per interval. The default of 3 would
mean 33 minutes on an 11m heartbeat and over a day on a 7h one.
More importantly the heartbeat window ALREADY encodes the tolerance - an 11m
window on a 5-minute push is exactly "one missed push forgiven" - so stacking a
threshold of 3 triples a tolerance that was already chosen. Hence:
push/heartbeat endpoints failure-threshold 1
pulled, 5m (public) failure-threshold 3 (= 15 minutes)
pulled, 6h/24h (dns, domain) failure-threshold 1
── The Signal transport ────────────────────────────────────────────────────
roles/signal_api runs signal-cli-rest-api on the observability host, pinned by
digest, MODE=native.
It publishes NO PORTS. The API has no authentication of any kind - anything that
reaches it can send messages as you and read your Signal. Gatus talks to it over
a shared docker network by service name, which is also WHY the network exists:
Gatus runs in a container, so the host's loopback is unreachable from it and a
port published on 127.0.0.1 would not have worked.
MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs
Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary
per request, and alerts are rare enough that startup cost per alert is the right
trade.
Monitored - Gatus polls /v1/health over the same network path the alerts take,
so it proves the delivery route rather than mere container liveness. Deliberately
NOT backed up: the data directory holds Signal private keys and the recovery
path is to link the device again from the phone.
Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert.
If either is down, Signal is precisely what cannot deliver the alert.
── Four traps hit while linking, all now in the role README ────────────────
* /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while
the binary itself emits a perfectly good URI. Not worked around by switching
MODE, which would put a JVM in the path of every alert permanently.
* The data dir must be owned by uid 1000, not root. A root-owned 0700 dir
cannot be traversed by the container user, so linking silently never
completes and /v1/accounts returns "Failed to read local accounts list".
* `docker exec` runs as ROOT while the service runs as uid 1000, so without
--config the account is written to /root/... on the container's ephemeral
layer. It reports success and is destroyed on the next recreate.
* The phone reporting "network error" was IPv6: chat.signal.org resolves to
dualstack AAAA records first, the container has no IPv6 address, and this
host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed
with a mounted gai.conf that prefers IPv4.
Verified: provider loads (configuredProviders=[signal]), a test message was
delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
watchtower is being destroyed. Removed from [vps], with its host_vars, its push
token, and the six Gatus endpoints that referenced it (liveness, disk, two
systemd services, the ntfy DNS record and the ntfy HTTP check).
ntfy went with it - it ran nowhere else - so services/ntfy is deleted,
subdomains.ntfy and ntfy_topic are gone from group_vars, and the ntfy playbook
is out of site.yml. ntfy_topic already had no readers: the three infra/4xx plays
that used it were deleted when their checks were superseded.
Two things this exposed.
services/ntfy/deploy_ntfy_playbook.yml was pointing at the WRONG MACHINE. It
said `hosts: observability`, which resolves to the host `monitoring`
(64.226.70.190) - but ntfy ran on watchtower, and ntfy.contrapeso.xyz pointed
there. Running it would have installed ntfy on the new VPS. Moot now, but it is
the same stale-identity failure as the rest: the group meant watchtower when the
play was written, and nobody revisited it when the group changed. Watchtower was
in [vps] and NO role group at all, while running caddy, ntfy and Uptime Kuma -
nothing in the repo managed any of it.
More seriously: ntfy-emergency-app on vipy (avisame.contrapeso.xyz) sends its
notifications to https://ntfy.contrapeso.xyz, topic "emergencia". Destroying
watchtower breaks it, and it is an EMERGENCY notifier - it would fail silently
at exactly the moment it matters. That is NOT resolved here, deliberately:
standing ntfy up elsewhere, pointing at ntfy.sh, or retiring the app are all
decisions, not cleanups.
What this change does is make the break impossible to miss. The URL was derived
from subdomains.ntfy, so deleting that would have turned it into an undefined
variable buried in a template. It is now an explicit ntfy_service_url in the
app's own vars, still holding the old value, with the three options written
above it. The ntfy credentials stay in the vault because that app still needs
them - the vault was restored from HEAD and only watchtower's push token
removed, rather than re-handling the plaintext.
Verified: no reference to watchtower or its IP anywhere in the repo; Gatus down
from 91 to 85 endpoints, 85 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Domains are now a list in group_vars/all (monitored_domains) rather than the
single root_domain hardcoded in the playbook, so adding one is a line of data
instead of a change to the task.
arbret.com resolves to prd-arbret (167.99.242.62) and its RDAP record exposes
an expiry of 2027-02-18, so the check reads real data rather than silently
passing on a missing field - verified against rdap.verisign.com before wiring
it up.
Both domains checked daily with two weeks of runway, because registration
renewal is a manual act at the registrar and losing a domain is not recoverable
the way losing a host is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four more check types, 42 endpoints, taking the estate from 39 to 81.
── systemd services (infra/401) ────────────────────────────────────────────
Every unit we deploy, checked every 5 minutes with a 16-minute heartbeat.
This closes the gap that let a real bug run unnoticed earlier today: a backup
script left forgejo, lnbits, headscale and memos stopped, and NOTHING caught
it. The dumps exited 0, the artefacts were correct, the deploy said failed=0,
and liveness only proves the HOST is up - not that anything on it serves.
One endpoint PER UNIT but only ONE timer per host: the check iterates that
host's units and pushes a result for each, the way check-backups.sh reports per
source. Four units on vipy would otherwise mean four scripts, services and
timers. A single host-level red light would also say "something on vipy is
down" without saying which, which is not the question anyone has.
Keys are host-qualified because unit names collide - caddy runs on four
machines. Which units a host runs lives in host_vars/<host>/monitored_services,
because "what runs here" is a property of the machine, the same reasoning as
the cross-host ports.
nut-driver-enumerator is deliberately excluded: it is a oneshot generator that
is `enabled` but always `inactive`, so it would report down forever. Checked
live before excluding it.
── domain, DNS and public endpoints (infra/402) ────────────────────────────
The first checks that PULL rather than push, and that is the right way round:
all three are about how the outside world sees us, so they must be measured
from outside. Nothing is installed anywhere - no script, no timer, no token.
They also have no heartbeat, because a heartbeat answers "did the reporter
report"; when Gatus does the checking itself, failure is immediate.
domain 1 endpoint, 24h, [DOMAIN_EXPIRATION] > 336h (two weeks)
dns 11 endpoints, 24h, [DNS_RCODE] == NOERROR and [BODY] == the IP
public 14 endpoints, 5m, 11 HTTPS + 3 TCP
Expected A records are derived from inventory (hostvars[host].ansible_host),
not written down again. The estate's recurring bug is an address recorded in a
second place and left behind when the machine moved; asserting against
inventory means a renumbered box is one edit, not two.
Expected HTTP status was checked live per site rather than assumed. Two return
401 - the Gatus dashboard and the DATUM dashboard, both behind basic auth - and
that is what is asserted: expecting 200 there would go green precisely when the
auth broke. headscale asserts /health rather than /, which is a 404 by design.
Every HTTPS check also carries [CERTIFICATE_EXPIRATION] > 168h, which is free
on an endpoint already being polled and catches a renewal that silently stops.
Two things learned the hard way, both now in comments:
* A domain-expiry endpoint needs a URL SCHEME. Gatus derives the endpoint type
from the prefix (endpoint.Type()), so a bare "contrapeso.xyz" is UNKNOWN and
the whole config is rejected. It is https:// plus a DOMAIN_EXPIRATION
condition and no status assertion, so the registrar's parking page at the
apex is irrelevant. Upstream also enforces a 5m minimum interval for that
placeholder, because it uses a free whois service.
* That rejection proved skip-invalid-config-update was worth adding. Gatus
logged "the configuration file was updated, but it is not valid, the old
configuration will continue being used" and kept running. Without it the
reload path calls panic() and one malformed contributed file takes the
monitor down.
Verified: 15/15 service endpoints UP; 26/27 public-facing UP. The single DOWN is
public_memos, correctly - memos-box is powered off, and the condition result
reads [STATUS] (502) == 200. memos-box and arbret-staging-box were shut down
deliberately from the Proxmox UI to test the liveness endpoints; their endpoints
stay registered and will go green when the VMs return.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
