Nothing in the repo pushes to, authenticates against, or is gated by Uptime
Kuma any more.
── The sixth instance of the banner bug ────────────────────────────────────
memos had `Restart memos` guarded by `uptime_kuma_enabled`, because the
deprecation banner was placed immediately above it and swept it in. It is a
HANDLER, so every memos config change since 2026-09-11 applied to disk and
silently never restarted the service. Ungated.
That is the same failure found in forgejo-runner's self-assert, phoenixd's timer
enable, mempool's three timer enables, fulcrum's restart handler and bitcoind's
restart handler. Every guard was read and asked "monitoring or deployment?"
before being deleted, which is the only reason this was caught.
── What was removed ────────────────────────────────────────────────────────
30 uptime_kuma_enabled guards across 7 unconverted service playbooks, and
the 29 Kuma monitor-creation tasks they gated (embedded Python that drove
the Kuma API, temp credential files, cleanup)
7 dead uptime_kuma_api_url definitions
7 stale DEPRECATED banners
uptime_kuma_enabled and subdomains.uptime_kuma from group_vars/all
healthcheck_push_urls from the vault - 30 push tokens
services/ntfy/setup_ntfy_uptime_kuma_notification.yml -> archive/
The explanatory comments in the six converted roles are KEPT on purpose. They
record why a handler is ungated, and deleting the explanation invites someone
to helpfully re-add the guard.
── The probes moved rather than died ───────────────────────────────────────
Eight per-service health checks were still pushing to Kuma. They are not
superseded by infra/401: that answers "is the unit running", these answer "does
the service actually respond" - an RPC call to bitcoind, a TCP connect to
Fulcrum's Electrum port, an HTTP fetch from Mempool's backend. A process can be
perfectly `active` and useless.
So they were repointed, not deleted. Gatus external endpoints take a POST with
a bearer token and success=true|false where Kuma took a GET with ?status=up, so
report() now maps up/down to true/false internally and no call site changed.
Registered by infra/403 as the `probe` group, one token per host.
Two bugs fixed while in there:
* forgejo-runner's check only ever reported SUCCESS - it exited before pushing
when the runner was down, so a failure was invisible until the heartbeat
window expired. Reporting the failure is the entire point of a check.
* All six healthcheck .service units were mode 0644 and now carry a bearer
token. They are 0600.
Verified: 91 endpoints, 91 UP, 0 DOWN. Every probe triggered by hand and
confirmed arriving. Zero Kuma URLs left in the vault, zero live references in
any playbook or role.
Still standing, deliberately: the Kuma container on watchtower, its Caddy vhost,
and the uptime.contrapeso.xyz DNS record. Turning the service off is a separate
decision from removing the code that talked to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four ports were the only entries in services_config.yml with a real
justification: each is read twice, by the role that deploys the service on its
own box AND by a socket-proxy or Caddy play that runs on the EDGE host and
publishes it. A role default is invisible to that second play.
But the shape was wrong in two ways. The file had to be named in vars_files: by
30 plays - opt-in configuration that someone will eventually forget - and five
role defaults silently interpolated service_settings.*, so bitcoin_knots,
fulcrum, datum_gateway and mempool were not self-contained: using any of them
without that one vars_file entry broke it.
Each port now lives in host_vars/<owning box>/main.yml:
host_vars/knots_box_local/main.yml bitcoin_p2p_port, datum_gateway_api_port,
datum_gateway_stratum_port
host_vars/fulcrum_box_local/main.yml fulcrum_ssl_port
host_vars/mempool_box_local/main.yml mempool_frontend_port
host_vars auto-loads and outranks role defaults, so the owning role picks the
value up with no vars_files at all, and the edge play reads the same single
definition as hostvars['<host>'].<name>. The role defaults keep the protocol
standard (8333, 50002, ...) so each role still works standalone, with the live
deployment's value in host_vars winning.
Also fixed a fourth copy of an inventory identity: the mempool Caddy play had
"mempool-box:{{ ... }}" hardcoded in the upstream. It now derives the host from
hostvars['mempool_box_local'].ansible_host, so inventory is the only place any
box's name is written down.
services_config.yml is deleted, with 25 more vars_files entries across 19
playbooks. Between this and the previous commit, 87 vars_files entries are gone
and every variable in the repo now comes from group_vars/all, host_vars,
inventory, a role default, or that service's own *_vars.yml.
Verification: an edge-host probe resolves all eight ports and hostnames to
byte-identical values to the ones services_config.yml used to supply. Each
owning host resolves its own port through host_vars. All 37 playbooks'
--list-tasks output is unchanged. The four edge plays that consume these values
all check-diff changed=0 - the socket-proxy and Caddy units on vipy are
byte-identical, which is the direct proof the rewiring landed on the same
values. fulcrum and datum-gateway check-diff exactly as before (ok=28/changed=1
and ok=15/changed=1, both the known timer re-arm).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three files existed only as second copies of things group_vars/all already
auto-loads, and 34 playbooks named them in vars_files: - which outranks
group_vars, so the copies won. The day someone edited one and not the other,
those plays would silently keep the stale value. infra_vars.yml was already
drifting: group_vars/all/main.yml had grown age_backup_recipient and
backup_pull_public_key that it lacked.
infra_vars.yml - a strict subset of group_vars/all/main.yml
infra_secrets.yml - decrypts byte-identical to group_vars/all/vault.yml
infra_secrets.yml.example - documented Uptime Kuma credentials as the reason
the file exists, which stopped being true
Deleted, along with 62 vars_files entries across 34 playbooks (12 of which
named ../../group_vars/all/main.yml directly - same defect, a vars_files entry
duplicating an auto-loaded file at higher precedence than the file itself).
Checked before touching anything: infra_secrets.yml was listed LAST in 10 plays,
after services_config.yml, so removal would flip precedence if the two shared a
key. They share none, and neither does services_config.yml with
group_vars/all/main.yml, so the removal is provably inert.
services_config.yml was the last one standing. It held four unrelated things:
caddy_sites_dir - an identical copy of roles/caddy_site/defaults/.
Deleted; the role default is now the only one.
*.tailscale_hostname (x3) - a THIRD copy of each box's identity, which
inventory.ini already holds as ansible_host.
Deleted. Edge plays now read
hostvars['<host>'].ansible_host - verified an
edge play resolves that with nothing loaded and
the other host in no play. Three copies of one
name is how bitcoin_rpc_host ended up labelled
"knots_box" while pointing at fulcrum-box.
subdomains, ntfy topic, - genuinely global: their readers span managed,
headscale namespace monitoring, vpn_control and edge, so no single
group covers them. Moved to group_vars/all/main.yml
where they auto-load. The ntfy_topic and
headscale_namespace indirection through
service_settings collapses to the global name.
the four cross-host ports - the only entries with a real justification.
Left in place; they move in the next commit.
Also dead, all Uptime Kuma residue or duplication:
phoenixd_monitor_name, forgejo_runner healthcheck_timeout_seconds/retries,
fulcrum_tailscale_hostname, and bitcoin_knots_version - the last being a
v-prefixed copy of bitcoin_knots_version_short that nothing read, two
hand-maintained copies of one version string.
Corrected a false comment: services_config.yml claimed the uptime_kuma subdomain
"no longer resolves to anything". It resolves to 164.92.239.72 and answers HTTP
302, and 11 playbooks still template it. Same wrong premise as PLAN_3.
Verification: all 37 playbooks' --list-tasks output is byte-identical before and
after. A probe resolving all 22 values services_config.yml used to supply returns
21 identical and one intended deletion (caddy_sites_dir, now role-only - confirmed
the role still resolves it: "Ensure Caddy sites-enabled directory exists" comes
back ok against the real path). memos check-diff identical before and after.
Syntax passes on every playbook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
552-line playbook becomes 18 lines plus a 411-line role
(install/service/healthcheck phases, four templates, two handlers).
phoenixd_vars.yml is deleted; its content is the role's defaults.
Task-list diff vs the old playbook shows ONLY the eight Uptime Kuma tasks
removed - everything else identical and in the same order.
phoenixd holds a Lightning node, so the run was checked against a pre-flight:
before: channel 6c25fa83..., balanceSat 1550723, capacitySat 3114830,
blockHeight 966692, active since 2026-09-02
after: identical, and still active since 2026-09-02 - it did NOT restart
`Create phoenixd systemd service` came back unchanged, which is what proves the
template reproduces the live unit byte-for-byte. changed=3 was the health check
script, its unit (Environment rename), and the timer restart. Second run:
changed=0.
Two things the conversion fixed, both symptoms of the deprecation banner having
been applied to contiguous blocks rather than to individual tasks:
- The health check logged "ERROR: UPTIME_KUMA_PUSH_URL not set" on every fire -
about 1,400 times a day - because its Environment= was emptied at
decommissioning. The exit code was still correct so nothing was broken, but it
is exactly the kind of noise that trains you to ignore a log. An unset push
URL is now normal and silent.
- `Enable and start phoenixd health check timer` was guarded by
uptime_kuma_enabled and so had not run since the decommissioning, while the
timer itself was still live on the host from before. Ansible had quietly
stopped managing something that was still running. Ungated.
Noted, not changed: seed.dat is mode 0644 on the host. That is phoenixd's own
doing, but it is a Lightning seed and worth tightening.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All five close-out greps return nothing: no sites-enabled handling outside
roles/, no `systemctl reload caddy`, no caddy_sites_dir self-reference, no
inline proxy unit writes. 37 playbooks syntax clean. The 14 Caddy site files
and 6 proxy units on the hosts are byte-identical to the Stage 0 baseline.
Seven play names still said "on vipy" while the play targeted a group. Renamed
to "on the edge host" - the last place a play claimed a hostname after Plan 2.
Documented the four vhosts in /etc/caddy/sites-enabled that no playbook writes
(uptime-kuma, arbretstaging, bitcoininfra, scriberr) in the caddy_site README.
None deleted.
uptime-kuma.conf was going to be deleted as dead config. It is not dead: the
louislam/uptime-kuma container is STILL RUNNING on watchtower - created
2026-02-07, restart=unless-stopped, healthy - and uptime.contrapeso.xyz returns
302, not the 502 a dead backend would give. The "decommissioning" retired the
Ansible code and the vault credentials, not the service. PLAN_3 claimed "the
tokens died with the server"; that is corrected there.
The Caddyfile.* backups are kept: one per host, Nov-Dec 2025, not churning, and
the only record of each Caddyfile before the import line was added.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the credentials would otherwise break these playbooks mid-deploy: they
template uptime_kuma_password with no assert to stop them first. 100 tasks are
now guarded by uptime_kuma_enabled (false), so deployments run normally and the
monitoring sections skip. A further 28 tasks were already self-guarding on
monitor_setup/push_url being defined; verified that a skipped task's registered
variable makes those skip cleanly rather than error.
The blocks are kept on purpose — the health-check logic is the durable part and
should be rewired to whatever replaces Uptime Kuma.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
