Five checks, 27 endpoints, replacing what Uptime Kuma used to watch:
is it up every 5min, all hosts
is disk full daily, all hosts
is CPU hot every 5min, nodito
is ZFS broken daily, nodito
is UPS online every 5min, nodito
Two roles, kept separate so neither knows about the other - they meet at a URL
and a token, the same way caddy_site and each service meet at a vhost:
roles/gatus_endpoint runs on the observability host, writes ONE file into
/opt/gatus/config/endpoints/. Gatus merges every *.yaml
there and appends lists, so callers compose without
coordinating.
roles/healthcheck runs on the monitored host: a check script, a systemd
service, a timer, and an optional push. Ships a library
of check bodies under templates/checks/.
Everything PUSHES. Gatus never reaches out, which matters because nodito and its
VMs are behind NAT, and because four of the five checks are internal state with
no pollable surface at all. Liveness pushes too, deliberately: a heartbeat proves
the host is running AND can reach the internet, where an ICMP probe from one
vantage point only proves it answers pings from there. And since Gatus alerts
when a heartbeat window expires, a check that stops running raises the alarm by
itself - a dead timer looks exactly like a dead host, which is the correct
reading.
One bearer token per host, generated straight into the vault and never printed.
A token only writes results for its own host's endpoints, so a compromised host
can lie about itself, which it could do anyway.
Three things learned from the source that shaped this:
* Gatus polls its own config every 30s and reloads
(main.listenToConfigurationFileChanges), so gatus_endpoint needs no restart
handler - writing the file IS the deploy.
* ...but on a reload it panics if the new config fails to parse, unless
skip-invalid-config-update is set. Endpoint files are contributed by other
playbooks, so one malformed file would take the monitor down at the worst
possible moment. Now set.
* The push URL uses a key Gatus computes, not the name you write:
sanitize(group) + "_" + sanitize(name), lowercased with / _ . , space # + &
replaced by "-" (config/key/key.go). So knots_box_local is
knots-box-local in the URL. The playbook derives it rather than hand-writing.
storage: maximum-number-of-results 900, up from upstream's 100. Gatus bounds the
database by COUNT and trims inline on insert, so there is no retention job and
no way to fill a disk - but history depth is then a function of check frequency,
and 100 results at a 5-minute interval is 8 hours. 900 is ~3 days of liveness
and ~2.5 years of the daily disk check. The uptime table is separate and its
30-day retention is hard-coded upstream.
A bug worth recording: the first deploy shipped five scripts that all died with
"syntax error: unexpected end of file". Jinja strips an included template's
trailing newline and trim_blocks then eats the newline after {% endif %}, so the
closing brace of check() landed on the same line as the body's last statement -
`return 0}`. Every check was broken and the deploy still reported failed=0,
because the role's "run once" task has failed_when: false and reports the result
as a debug message nobody read. The blank line that fixes it is now load-bearing
and commented as such.
Verified by triggering every unit by hand rather than waiting on timers: all
checks exit 0 on all hosts, and Gatus shows 26 UP / 1 DOWN. The one DOWN is
liveness_watchtower, which is honest - that host currently refuses SSH (TCP
connects, no banner exchange) and is excluded from this deploy. It is also the
box still running Uptime Kuma.
Known waste, not yet fixed: healthcheck installs its dependencies per CHECK
rather than per HOST, so apt runs 29 times estate-wide for a curl that is
already present, and daemon_reload runs 4x per host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First step of replacing Uptime Kuma and ntfy. Gatus runs on the new
observability VPS, fronted by Caddy at status.contrapeso.xyz.
Deployed as the upstream container image, not built from source. The role did
build from source first - their Dockerfile is a bare `CGO_ENABLED=0 go build`,
the Vue dashboard is compiled in via `//go:embed static` in web/static.go, and
CGO can stay off because the sqlite driver is pure-Go modernc.org/sqlite - but
that produces a binary upstream never ran, and it meant compiling the AWS SDK
and gRPC on the smallest box in the estate. That load was heavy enough that
unrelated Ansible tasks timed out while it ran. The cost of the container is a
daemon on the machine whose job is to notice when everything else breaks; that
trade is made deliberately and is written down in the role README.
Pinned by DIGEST, not tag. A tag is mutable - v5.36.0 can be repushed - so
pinning it alone is a weaker promise than it looks:
gatus_image: "ghcr.io/twin/gatus@sha256:c5f210d0..."
`docker compose pull` now either fetches exactly the reviewed image or fails.
gatus_version is kept beside it only so a human can read the release; the two
move together.
The image is FROM scratch, so it has no /etc/passwd and its default user is
root. The container runs as 10001:10001 with the host data dir owned to match,
plus read_only, cap_drop ALL, and no-new-privileges. NET_RAW is added back only
when gatus_allow_icmp, so the capability for icmp:// checks is a visible grant
rather than something inherited from running as root.
Config is a DIRECTORY, not a file. Gatus merges every *.yaml under
GATUS_CONFIG_PATH - maps deep-merge, lists append - so the role owns
00-base.yaml (web, storage, ui, alerting, security) and each service will drop
its own file into endpoints/, the same shape as caddy_site. A primitive defined
twice is ambiguous and upstream refuses it, so anything that is not a list lives
in the base file and nowhere else.
Two bugs the deploy caught:
* Gatus panics on a config with no endpoints ("configuration should contain at
least one endpoint or suite"), so "install now, add endpoints later" is not
a valid state. The role ships endpoints/00-self.yaml checking its own
/health. Less circular than it looks: it proves the directory merged, the
listener serves, and storage accepted a write.
* web.address was carried over from the systemd design as 127.0.0.1. Inside a
container that is the CONTAINER's loopback, which docker-proxy cannot reach
- gatus came up healthy, self-check passing, while every connection to the
published port was refused. It now always binds 0.0.0.0 inside the
container; the isolation comes from publishing to 127.0.0.1 on the host.
Auth is done at the edge, NOT with Gatus's own security.basic. Reading
api/api.go, that middleware protects exactly four routes - the statuses
endpoints. Everything else is registered on the unprotected router, including
/api/v1/config, every badge, and /api/v1/endpoints/:key/uptimes/:duration and
.../response-times/:duration/history, which return real data to anyone who can
guess a key ("<group>_<name>"). Verified against the live instance: all seven
routes returned 200 unauthenticated, and /uptimes/24h returned "1.000000".
So the vhost uses caddy_site_body with a path carve-out rather than
caddy_site_basic_auth, which has no way to exempt a path. The external-endpoint
push API must NOT sit behind basic auth: it authenticates with
`Authorization: Bearer <token>`, and basic auth wants the same header. It is not
unauthenticated - the handler 401s on a missing prefix, an empty token, or a
token that does not match that endpoint's own.
Verified end to end. All seven previously-open routes now 401. The push path
distinguishes cleanly: POST with no auth gets Gatus's own "invalid Authorization
header" with NO WWW-Authenticate; POST with a bogus Bearer gets 404 (key looked
up, no external endpoints yet); GET on the same path gets Caddy's 401 with
WWW-Authenticate: Basic, so the exemption is scoped to POST alone. The
self-check still passes because it polls localhost inside the container and
never traverses Caddy.
The host itself was rebuilt from scratch: 01 (ok=9 changed=8), 02 (ok=12
changed=6), 910_docker (--limit, since that playbook still wrongly claims all of
`managed` needs Docker), caddy (ok=13 changed=8), gatus (ok=18 changed=2).
Not done here: gatus_alerting is still {} - valid, and every condition is
evaluated and recorded, there is just nowhere to shout until a provider is
chosen to replace ntfy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These two are host-specific by design - nodito is a pet, not cattle - so they
stay playbooks rather than becoming roles. But they had rotted.
De-Kuma, following the pattern of the six service roles:
Both plays opened with an assert on uptime_kuma_username/password, which were
removed from the vault, so both failed before doing anything. Dropped that,
the two embedded Python monitor-creation scripts, and their /tmp cleanup.
Kept every check, threshold and systemd timer - those are the durable part.
Reporting is now generic: `healthcheck_push_url` goes into the unit as
Environment=HEALTHCHECK_PUSH_URL and the script reads ${HEALTHCHECK_PUSH_URL:-},
treating empty as normal rather than an error. The exit code is the real
answer; systemd keeps it. Both scripts now also report status=down on failure
instead of only going silent. Live push URLs harvested into the vault so
nothing observable changes for ZFS.
Three live bugs found while check-diffing:
1. 32_zfs would have DE-REGISTERED the Proxmox storage. `pvesm remove` was
gated on the storage existing and `pvesm add` on it NOT existing - mutually
exclusive - so a real run removed the proxmox-tank-1 entry backing every VM
and never put it back. It would also have dropped `mountpoint /var/lib/vz`,
which the live entry has and `pvesm add` does not set. Registration is now
add-only.
2. zfs_disk_1 named ata-...WX11TN0Z, a disk no longer in the machine. The live
mirror is WX120LHQ + WX11TN2P; a leg was replaced and the repo never caught
up. Inert behind `when: zfs_pool_exists.rc != 0`, but wrong on any
disaster-recovery run. This is the seventh instance of an identifier
written down once whose hardware later moved.
3. 34_nut has NEVER been applied to nodito - no /etc/nut file carries the
"Managed by Ansible" marker; they were written by hand in January 2026. The
vault held the literal CHANGE_ME_TO_SECURE_PASSWORD, so applying it would
have overwritten a working upsd/upsmon auth pair with a placeholder and
restarted NUT, leaving the hypervisor's UPS unable to trigger a clean
shutdown on mains loss. The Kuma assert was the only thing stopping that,
so removing it without a replacement would have armed the gun: there is now
an explicit assert that refuses to run on the placeholder. The real
password is in the vault and `Configure upsd users` check-diffs clean.
Templates reconciled with the live files first, so applying 34_nut is close to
a no-op: added `maxretry = 3` to ups.conf and OFFDURATION / RBWARNTIME /
NOCOMMWARNTIME / FINALDELAY plus quoted POWERDOWNFLAG to upsmon.conf. The only
substantive additions left are the NOTIFYMSG/NOTIFYFLAG syslog lines.
/usr/local/bin/ups-heartbeat.sh on the box is an orphan - mode 0644, not
executable, referenced by no unit and no cron entry - but its push token
belongs to a monitor that still exists and answers, so that monitor has had no
heartbeat since January. Harvested as healthcheck_push_urls.ups; applying this
play is what will finally feed it.
Finish the host_vars migration:
infra/nodito/nodito_vars.yml was byte-identical to host_vars/nodito/main.yml.
Deleted it, moved nodito_secrets.yml to host_vars/nodito/vault.yml, and
stripped the dead vars_files entries from all three playbooks.
Extract the 13 inline `content: |` blocks to infra/nodito/templates/, pulled via
a YAML load rather than retyped. 681+582 lines become 313+344 plus templates.
Ownership parity against HEAD checked mechanically: no owner/group/mode drift on
any surviving task.
Neither playbook has been applied. ZFS play 2 check-runs failed=0 with two
changes: the script rewrite (the deployed one is a hand-edited DEBUG VERSION
with `set -x`) and the Environment line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
802-line playbook becomes 68 lines (three plays: the role, the Caddy dashboard,
the Stratum socket proxy) plus a 345-line role. datum_gateway_vars.yml is
deleted; its content is the role's defaults.
Verified after a real run with zero miners connected: datum-gateway restarted
cleanly onto the reformatted config, deployed config.json semantically identical
to what was there (pool_address bc1qvrj3g84..., pool_pass_* false, ports
unchanged), health check timer firing, and the Knots side untouched - bitcoind
still up since 2026-08-19 with blocknotify intact.
TWO PIECES OF DRIFT WHERE THE NODE WAS RIGHT, both confirmed with the operator:
- datum_mining_address: the vault held bc1qdse9dsg... while the node had been
mining to bc1qvrj3g... since 2026-08-08. This is WHERE BLOCK REWARDS ARE PAID.
And unlike fulcrum and bitcoin-knots, the `Restart datum-gateway` handler here
was never gated, so the stale value would have applied immediately rather than
sitting inert on disk.
- pool_pass_workers / pool_pass_full_users: false on the node, true in the vars
file.
Both corrected in the vault and role defaults with notes recording why.
Comparing this config needs semantics, not text: the live file is single-line
JSON and the template renders pretty-printed, so a textual diff is pure noise.
Rendering it and comparing parsed JSON is what surfaced both differences.
config.json carries bitcoind.rpcpassword and api.admin_password, and --diff
prints rendered content - so `--check --diff` put them on the terminal. The task
now sets diff: false by default (-e datum_reveal_config=true to opt in). Those
two should be rotated.
I also mis-reported pool_pass_workers/pool_pass_full_users as exposed credentials
because my masking matched "pass" in the key name. They are BOOLEANS, and
mining.pool_address is a Bitcoin address, public by nature. Only the two real
passwords above were exposed.
`Configure cmake build` and `Compile datum_gateway` are bare command: tasks with
no changed_when, so they recompile on every run. The build is reproducible -
Install datum_gateway binary sees identical content and leaves the installed
binary's timestamp alone - but it is wasted work each time. Documented as the
idempotent floor.
Ownership parity checked mechanically against `git show HEAD:` keyed by task
name: 7/7 match, 9 Kuma tasks dropped.
This completes Plan 6 Stage 2: all six services in the list are roles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
892-line playbook becomes 40 lines plus a role with install/build/configure/
service/healthcheck phases, five templates and one handler. bitcoin_knots_vars.yml
is deleted; its content is the role's defaults.
Verified after a real run: bitcoind still active since 2026-08-19 (NO restart),
chain at 966844 blocks / 875 GB, DATUM config intact, dbcache still 200, health
check timer firing again. changed=3, all health-check. vipy changed=0.
⚠ THE BIG ONE: the playbook would have deleted the mining integration.
bitcoin.conf on the node carries a section that was hand-added and was missing
from the template entirely:
blockmaxsize=3985000
blockmaxweight=3985000
blocknotify=killall -USR1 datum_gateway
maxmempool=1000
blockreconstructionextratxn=1000000
blocknotify is how datum_gateway learns a new block landed. Running the old
playbook would have stripped all of it and solo mining would have carried on
against a stale template - a silent failure that costs money rather than raising
an error. Also dbcache 200 -> 3528 (hand-tuned down; the calculation wants 90% of
RAM) and logging moved off the file. All now reconciled, dbcache behind
bitcoin_dbcache_mb_override.
bitcoin-knots and datum-gateway are ONE SYSTEM. Noted in the README.
AND MY OWN FIX MADE IT MORE DANGEROUS. The `Restart bitcoind` handler was guarded
by uptime_kuma_enabled, so it had been inert: bitcoin.conf and the systemd unit
both notify it and neither could restart anything - a config change applied to
disk, reported success, and never took effect. Ungating that is right, but it
converts "wrong config sitting inertly on disk" into "node restarted onto a
config that breaks mining". The ungating had to land WITH the template
reconciliation, not before it.
It also raises the bar permanently: any residual template/live difference now
restarts a Bitcoin node on every run. Four rounds of --check --diff to reach
changed=0 - the DATUM section, an explanatory comment that was rendering into the
deployed config (now a {# #} Jinja comment), a "# Pruning (optional)" comment the
live file had, and one trailing blank line.
The build path is 32 tasks all guarded by `not bitcoind_binary_exists.stat.exists`,
so a converged host skips the 30-60 minute compile and both `state: absent`
deletions. Those target /opt/bitcoin-knots/{source,bitcoin-<version>}; the chain
is in /mnt/knots_data and is never touched. Signature-verification tasks copied
verbatim.
The health check timer had last fired 2026-08-09 while reporting active/enabled -
same OnBootSec + OnUnitActiveSec dead chain as fulcrum. The role runs the check
once after enabling to supply the reference the timer schedules from.
Ownership parity checked mechanically against `git show HEAD:`, keyed by TASK
NAME rather than path - keying by path gave a false positive, because
bitcoin_knots_source_dir is created with ownership and later removed with
state: absent, so whichever task comes last wins and that differs between one
file and five. 13/13 match.
bitcoin_p2p_port and the tailscale hostname moved to services_config.yml for the
socket-proxy play on the edge host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
745-line playbook becomes 37 lines (the role, plus the Caddy play for the edge
host) and a 408-line role with docker/deploy/healthcheck phases and six
templates. mempool_vars.yml is deleted; its content is the role's defaults.
Three health checks are kept, not collapsed: Mempool is three moving parts and
knowing which one is down is the point. Each has its own script, unit, timer and
push_url, driven by a mempool_healthchecks list. The Uptime Kuma specifics are
gone - the embedded Python creating monitors over the API, the /tmp credentials
file, the push-URL file read back and parsed, three Environment= rewrites - and
the three live push URLs are preserved from the vault, so reporting is unchanged.
`Enable and start health check timers` and `Display deployment status` were both
guarded by uptime_kuma_enabled despite being deployment tasks. Third service in
a row with that pattern: the deprecation banner was applied to contiguous blocks,
so anything sitting near the push plumbing was disabled with it. Ungated.
TWO OWNERSHIP PROBLEMS, different in kind:
- MINE: I wrote `owner: root` on docker-compose.yml where the original says
`owner: "{{ ansible_user }}"`. A straight violation of extract-mechanically-
change-nothing, caught only by reading the check-mode diff line by line.
Reverted to match the original.
- PRE-EXISTING, and dangerous: the playbook declared
`owner: "{{ ansible_user }}"` (1000) on the MariaDB data directory, which the
container owns as uid 999. Confirmed against `git show HEAD:` before
concluding it was not mine. It had drifted since the containers were created
and went unnoticed because the playbook had not been run since.
This was not academic. The first real run pulled a newer mariadb:10.11 and
recreated mempool-db; with the chown still in place MariaDB would have come
back to a data directory it could not write. The role now ensures the
directory exists and leaves ownership to the container. Verified after the
run: /opt/mempool/mysql is still 999:999 and all three containers are healthy.
This is a deliberate behaviour change, not part of the extraction. It is in
this commit rather than a follow-up because the faithful version was never
safe to run, so there was no intermediate state worth recording as verified.
mempool_frontend_port moved to services_config.yml: two hosts need it (this role
deploys the frontend, the Caddy play proxies to it from the edge host) and a role
default is invisible to the second play. caddy_site's parameter assert caught
this loudly - "'mempool_frontend_port' is undefined" - rather than silently.
Verified: check-mode diff clean apart from unavoidable check-mode artifacts;
first run ok=24 changed=5, zero failures; second run changed=2 - the two bare
`command:` tasks (pull, compose up) that have no changed_when and always report
changed. That is the idempotent floor. All three health checks report
ExecMainStatus 0 with their push URLs intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
409-line playbook becomes a 16-line playbook plus a 318-line role with phases
split across tasks/{prerequisites,install,configure,service,healthcheck}.yml and
four templates. forgejo_runner_vars.yml is deleted; its content is the role's
defaults.
Applies the Plan 6 Stage 0 decision: keep whatever determines whether the
service is healthy, drop the Uptime Kuma specifics, make the reporting point
pluggable. Gone from the role: the embedded Python that created monitors over
the Kuma API, the /tmp credentials file, token extraction, the systemd
Environment= rewrite, and 8 `when: uptime_kuma_enabled` guards. What remains is
the check itself, its log, the systemd unit and timer, and an honest exit code -
`systemctl is-failed forgejo-runner-healthcheck.service` now answers the
question with no monitoring system involved at all.
Reporting is one variable, healthcheck_push_url, empty by default. Any endpoint
that accepts an HTTP ping plugs in there. A pull-based monitor wants it left
empty and reads unit state instead.
PREMISE CORRECTION: Uptime Kuma is NOT dead. Plan 3 recorded "48 push timers
curling an endpoint that no longer answers" and Plan 6 said the check had
"nowhere to report to". Both wrong - 24+ push scripts across 11 hosts are
pushing successfully right now (HTTP 200). Only the Ansible code and the vault
credentials were decommissioned; the service never stopped. So the existing push
URLs were harvested into a vaulted healthcheck_push_urls dict and are preserved,
keeping this refactor behaviour-neutral. Retiring Kuma stays a deliberate act
rather than a side effect. PLAN_3 and PLAN_6 are corrected.
Verified:
- task-list diff vs the old playbook shows ONLY the five Kuma tasks removed,
everything else identical and in the same order
- first run ok=22 changed=1 (the rewritten health script); both systemd units
and forgejo-runner.service came back ok, so the templates reproduce the
previous files byte-for-byte
- second run ok=22 changed=0, fully idempotent
- still reports "Ping sent successfully (HTTP 200)" from a script containing
zero Uptime Kuma references
- the 4 skipped tasks are genuine already-configured guards, checked not assumed
Two things for the next service:
- import_tasks, not include_tasks. Dynamic includes are opaque to --list-tasks,
which is the primary verification tool here; the first attempt produced a
useless diff.
- `Assert runner is running` was guarded by uptime_kuma_enabled and so had not
run since the decommissioning. It is not monitoring, it is the deployment
checking its own work - the deprecation banner swept it up with the Kuma
plumbing, and a runner that failed to start was deploying "successfully" in
silence. Ungated now. The banner was applied to contiguous blocks, so read
every uptime_kuma_enabled guard and ask whether it is monitoring or deployment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes Stage 3. No hand-rolled Caddy plumbing remains anywhere:
`grep sites-enabled` outside roles/ returns nothing, and so does
`grep "systemctl reload caddy"`.
ntfy uses caddy_site_body for its plain-HTTP listener and @httpget redirect.
Verified ok/unchanged against watchtower; the one other changed task is a
pre-existing "Update APT cache".
datum-gateway keeps a whole-Caddyfile validate after the role call. The role
validates its own fragment, but only a whole-file validate catches a conflict
between two sites, and this playbook was the only one that ever had it. Its
two debug tasks that echoed command output are gone with the commands.
headscale is the one that mattered. Its playbook wrote
`reverse_proxy localhost:8080`, but spacey is actually running a /admin*
route in front of Headplane behind Caddy basic auth. Running that playbook
would have deleted the admin route and its auth - a hazard that predates this
work. It now renders the config that is really there, verified ok/unchanged
via --start-at-task (the play cannot reach Caddy in check mode: "Install
headscale package" fails because the .deb is not really downloaded, before
and after this edit alike).
Supporting changes for headscale:
- headscale_ui_password_hash added to infra_secrets.yml and the identical
group_vars/all/vault.yml, read from the live config on spacey. The vault
already had headscale_ui_username (= counterweight, confirmed) and
headscale_ui_password; I did not verify the password is the plaintext of
this hash.
- headplane_port added to headscale_vars.yml.
- The role's handler now sets become: true. Handlers do not inherit become
from the task that notified them, and this play runs become: no.
- The include uses `apply: become: yes`; `become:` on an include_role is
rejected outright.
All 14 site files on all 3 hosts still byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops uptime_kuma_username/password from infra_secrets.yml and its group_vars
copy and example, plus a dead push token in nodito_secrets.yml that nothing
referenced. They remain in git history — rotation is what actually retires them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
