watchtower is being destroyed. Removed from [vps], with its host_vars, its push
token, and the six Gatus endpoints that referenced it (liveness, disk, two
systemd services, the ntfy DNS record and the ntfy HTTP check).
ntfy went with it - it ran nowhere else - so services/ntfy is deleted,
subdomains.ntfy and ntfy_topic are gone from group_vars, and the ntfy playbook
is out of site.yml. ntfy_topic already had no readers: the three infra/4xx plays
that used it were deleted when their checks were superseded.
Two things this exposed.
services/ntfy/deploy_ntfy_playbook.yml was pointing at the WRONG MACHINE. It
said `hosts: observability`, which resolves to the host `monitoring`
(64.226.70.190) - but ntfy ran on watchtower, and ntfy.contrapeso.xyz pointed
there. Running it would have installed ntfy on the new VPS. Moot now, but it is
the same stale-identity failure as the rest: the group meant watchtower when the
play was written, and nobody revisited it when the group changed. Watchtower was
in [vps] and NO role group at all, while running caddy, ntfy and Uptime Kuma -
nothing in the repo managed any of it.
More seriously: ntfy-emergency-app on vipy (avisame.contrapeso.xyz) sends its
notifications to https://ntfy.contrapeso.xyz, topic "emergencia". Destroying
watchtower breaks it, and it is an EMERGENCY notifier - it would fail silently
at exactly the moment it matters. That is NOT resolved here, deliberately:
standing ntfy up elsewhere, pointing at ntfy.sh, or retiring the app are all
decisions, not cleanups.
What this change does is make the break impossible to miss. The URL was derived
from subdomains.ntfy, so deleting that would have turned it into an undefined
variable buried in a template. It is now an explicit ntfy_service_url in the
app's own vars, still holding the old value, with the three options written
above it. The ntfy credentials stay in the vault because that app still needs
them - the vault was restored from HEAD and only watchtower's push token
removed, rather than re-handling the plaintext.
Verified: no reference to watchtower or its IP anywhere in the repo; Gatus down
from 91 to 85 endpoints, 85 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Domains are now a list in group_vars/all (monitored_domains) rather than the
single root_domain hardcoded in the playbook, so adding one is a line of data
instead of a change to the task.
arbret.com resolves to prd-arbret (167.99.242.62) and its RDAP record exposes
an expiry of 2027-02-18, so the check reads real data rather than silently
passing on a missing field - verified against rdap.verisign.com before wiring
it up.
Both domains checked daily with two weeks of runway, because registration
renewal is a manual act at the registrar and losing a domain is not recoverable
the way losing a host is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four more check types, 42 endpoints, taking the estate from 39 to 81.
── systemd services (infra/401) ────────────────────────────────────────────
Every unit we deploy, checked every 5 minutes with a 16-minute heartbeat.
This closes the gap that let a real bug run unnoticed earlier today: a backup
script left forgejo, lnbits, headscale and memos stopped, and NOTHING caught
it. The dumps exited 0, the artefacts were correct, the deploy said failed=0,
and liveness only proves the HOST is up - not that anything on it serves.
One endpoint PER UNIT but only ONE timer per host: the check iterates that
host's units and pushes a result for each, the way check-backups.sh reports per
source. Four units on vipy would otherwise mean four scripts, services and
timers. A single host-level red light would also say "something on vipy is
down" without saying which, which is not the question anyone has.
Keys are host-qualified because unit names collide - caddy runs on four
machines. Which units a host runs lives in host_vars/<host>/monitored_services,
because "what runs here" is a property of the machine, the same reasoning as
the cross-host ports.
nut-driver-enumerator is deliberately excluded: it is a oneshot generator that
is `enabled` but always `inactive`, so it would report down forever. Checked
live before excluding it.
── domain, DNS and public endpoints (infra/402) ────────────────────────────
The first checks that PULL rather than push, and that is the right way round:
all three are about how the outside world sees us, so they must be measured
from outside. Nothing is installed anywhere - no script, no timer, no token.
They also have no heartbeat, because a heartbeat answers "did the reporter
report"; when Gatus does the checking itself, failure is immediate.
domain 1 endpoint, 24h, [DOMAIN_EXPIRATION] > 336h (two weeks)
dns 11 endpoints, 24h, [DNS_RCODE] == NOERROR and [BODY] == the IP
public 14 endpoints, 5m, 11 HTTPS + 3 TCP
Expected A records are derived from inventory (hostvars[host].ansible_host),
not written down again. The estate's recurring bug is an address recorded in a
second place and left behind when the machine moved; asserting against
inventory means a renumbered box is one edit, not two.
Expected HTTP status was checked live per site rather than assumed. Two return
401 - the Gatus dashboard and the DATUM dashboard, both behind basic auth - and
that is what is asserted: expecting 200 there would go green precisely when the
auth broke. headscale asserts /health rather than /, which is a 404 by design.
Every HTTPS check also carries [CERTIFICATE_EXPIRATION] > 168h, which is free
on an endpoint already being polled and catches a renewal that silently stops.
Two things learned the hard way, both now in comments:
* A domain-expiry endpoint needs a URL SCHEME. Gatus derives the endpoint type
from the prefix (endpoint.Type()), so a bare "contrapeso.xyz" is UNKNOWN and
the whole config is rejected. It is https:// plus a DOMAIN_EXPIRATION
condition and no status assertion, so the registrar's parking page at the
apex is irrelevant. Upstream also enforces a 5m minimum interval for that
placeholder, because it uses a free whois service.
* That rejection proved skip-invalid-config-update was worth adding. Gatus
logged "the configuration file was updated, but it is not valid, the old
configuration will continue being used" and kept running. Without it the
reload path calls panic() and one malformed contributed file takes the
monitor down.
Verified: 15/15 service endpoints UP; 26/27 public-facing UP. The single DOWN is
public_memos, correctly - memos-box is powered off, and the condition result
reads [STATUS] (502) == 200. memos-box and arbret-staging-box were shut down
deliberately from the Proxmox UI to test the liveness endpoints; their endpoints
stay registered and will go green when the VMs return.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
