Nothing in the repo pushes to, authenticates against, or is gated by Uptime
Kuma any more.
── The sixth instance of the banner bug ────────────────────────────────────
memos had `Restart memos` guarded by `uptime_kuma_enabled`, because the
deprecation banner was placed immediately above it and swept it in. It is a
HANDLER, so every memos config change since 2026-09-11 applied to disk and
silently never restarted the service. Ungated.
That is the same failure found in forgejo-runner's self-assert, phoenixd's timer
enable, mempool's three timer enables, fulcrum's restart handler and bitcoind's
restart handler. Every guard was read and asked "monitoring or deployment?"
before being deleted, which is the only reason this was caught.
── What was removed ────────────────────────────────────────────────────────
30 uptime_kuma_enabled guards across 7 unconverted service playbooks, and
the 29 Kuma monitor-creation tasks they gated (embedded Python that drove
the Kuma API, temp credential files, cleanup)
7 dead uptime_kuma_api_url definitions
7 stale DEPRECATED banners
uptime_kuma_enabled and subdomains.uptime_kuma from group_vars/all
healthcheck_push_urls from the vault - 30 push tokens
services/ntfy/setup_ntfy_uptime_kuma_notification.yml -> archive/
The explanatory comments in the six converted roles are KEPT on purpose. They
record why a handler is ungated, and deleting the explanation invites someone
to helpfully re-add the guard.
── The probes moved rather than died ───────────────────────────────────────
Eight per-service health checks were still pushing to Kuma. They are not
superseded by infra/401: that answers "is the unit running", these answer "does
the service actually respond" - an RPC call to bitcoind, a TCP connect to
Fulcrum's Electrum port, an HTTP fetch from Mempool's backend. A process can be
perfectly `active` and useless.
So they were repointed, not deleted. Gatus external endpoints take a POST with
a bearer token and success=true|false where Kuma took a GET with ?status=up, so
report() now maps up/down to true/false internally and no call site changed.
Registered by infra/403 as the `probe` group, one token per host.
Two bugs fixed while in there:
* forgejo-runner's check only ever reported SUCCESS - it exited before pushing
when the runner was down, so a failure was invisible until the heartbeat
window expired. Reporting the failure is the entire point of a check.
* All six healthcheck .service units were mode 0644 and now carry a bearer
token. They are 0600.
Verified: 91 endpoints, 91 UP, 0 DOWN. Every probe triggered by hand and
confirmed arriving. Zero Kuma URLs left in the vault, zero live references in
any playbook or role.
Still standing, deliberately: the Kuma container on watchtower, its Caddy vhost,
and the uptime.contrapeso.xyz DNS record. Turning the service off is a separate
decision from removing the code that talked to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four ports were the only entries in services_config.yml with a real
justification: each is read twice, by the role that deploys the service on its
own box AND by a socket-proxy or Caddy play that runs on the EDGE host and
publishes it. A role default is invisible to that second play.
But the shape was wrong in two ways. The file had to be named in vars_files: by
30 plays - opt-in configuration that someone will eventually forget - and five
role defaults silently interpolated service_settings.*, so bitcoin_knots,
fulcrum, datum_gateway and mempool were not self-contained: using any of them
without that one vars_file entry broke it.
Each port now lives in host_vars/<owning box>/main.yml:
host_vars/knots_box_local/main.yml bitcoin_p2p_port, datum_gateway_api_port,
datum_gateway_stratum_port
host_vars/fulcrum_box_local/main.yml fulcrum_ssl_port
host_vars/mempool_box_local/main.yml mempool_frontend_port
host_vars auto-loads and outranks role defaults, so the owning role picks the
value up with no vars_files at all, and the edge play reads the same single
definition as hostvars['<host>'].<name>. The role defaults keep the protocol
standard (8333, 50002, ...) so each role still works standalone, with the live
deployment's value in host_vars winning.
Also fixed a fourth copy of an inventory identity: the mempool Caddy play had
"mempool-box:{{ ... }}" hardcoded in the upstream. It now derives the host from
hostvars['mempool_box_local'].ansible_host, so inventory is the only place any
box's name is written down.
services_config.yml is deleted, with 25 more vars_files entries across 19
playbooks. Between this and the previous commit, 87 vars_files entries are gone
and every variable in the repo now comes from group_vars/all, host_vars,
inventory, a role default, or that service's own *_vars.yml.
Verification: an edge-host probe resolves all eight ports and hostnames to
byte-identical values to the ones services_config.yml used to supply. Each
owning host resolves its own port through host_vars. All 37 playbooks'
--list-tasks output is unchanged. The four edge plays that consume these values
all check-diff changed=0 - the socket-proxy and Caddy units on vipy are
byte-identical, which is the direct proof the rewiring landed on the same
values. fulcrum and datum-gateway check-diff exactly as before (ok=28/changed=1
and ok=15/changed=1, both the known timer re-arm).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three files existed only as second copies of things group_vars/all already
auto-loads, and 34 playbooks named them in vars_files: - which outranks
group_vars, so the copies won. The day someone edited one and not the other,
those plays would silently keep the stale value. infra_vars.yml was already
drifting: group_vars/all/main.yml had grown age_backup_recipient and
backup_pull_public_key that it lacked.
infra_vars.yml - a strict subset of group_vars/all/main.yml
infra_secrets.yml - decrypts byte-identical to group_vars/all/vault.yml
infra_secrets.yml.example - documented Uptime Kuma credentials as the reason
the file exists, which stopped being true
Deleted, along with 62 vars_files entries across 34 playbooks (12 of which
named ../../group_vars/all/main.yml directly - same defect, a vars_files entry
duplicating an auto-loaded file at higher precedence than the file itself).
Checked before touching anything: infra_secrets.yml was listed LAST in 10 plays,
after services_config.yml, so removal would flip precedence if the two shared a
key. They share none, and neither does services_config.yml with
group_vars/all/main.yml, so the removal is provably inert.
services_config.yml was the last one standing. It held four unrelated things:
caddy_sites_dir - an identical copy of roles/caddy_site/defaults/.
Deleted; the role default is now the only one.
*.tailscale_hostname (x3) - a THIRD copy of each box's identity, which
inventory.ini already holds as ansible_host.
Deleted. Edge plays now read
hostvars['<host>'].ansible_host - verified an
edge play resolves that with nothing loaded and
the other host in no play. Three copies of one
name is how bitcoin_rpc_host ended up labelled
"knots_box" while pointing at fulcrum-box.
subdomains, ntfy topic, - genuinely global: their readers span managed,
headscale namespace monitoring, vpn_control and edge, so no single
group covers them. Moved to group_vars/all/main.yml
where they auto-load. The ntfy_topic and
headscale_namespace indirection through
service_settings collapses to the global name.
the four cross-host ports - the only entries with a real justification.
Left in place; they move in the next commit.
Also dead, all Uptime Kuma residue or duplication:
phoenixd_monitor_name, forgejo_runner healthcheck_timeout_seconds/retries,
fulcrum_tailscale_hostname, and bitcoin_knots_version - the last being a
v-prefixed copy of bitcoin_knots_version_short that nothing read, two
hand-maintained copies of one version string.
Corrected a false comment: services_config.yml claimed the uptime_kuma subdomain
"no longer resolves to anything". It resolves to 164.92.239.72 and answers HTTP
302, and 11 playbooks still template it. Same wrong premise as PLAN_3.
Verification: all 37 playbooks' --list-tasks output is byte-identical before and
after. A probe resolving all 22 values services_config.yml used to supply returns
21 identical and one intended deletion (caddy_sites_dir, now role-only - confirmed
the role still resolves it: "Ensure Caddy sites-enabled directory exists" comes
back ok against the real path). memos check-diff identical before and after.
Syntax passes on every playbook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
802-line playbook becomes 68 lines (three plays: the role, the Caddy dashboard,
the Stratum socket proxy) plus a 345-line role. datum_gateway_vars.yml is
deleted; its content is the role's defaults.
Verified after a real run with zero miners connected: datum-gateway restarted
cleanly onto the reformatted config, deployed config.json semantically identical
to what was there (pool_address bc1qvrj3g84..., pool_pass_* false, ports
unchanged), health check timer firing, and the Knots side untouched - bitcoind
still up since 2026-08-19 with blocknotify intact.
TWO PIECES OF DRIFT WHERE THE NODE WAS RIGHT, both confirmed with the operator:
- datum_mining_address: the vault held bc1qdse9dsg... while the node had been
mining to bc1qvrj3g... since 2026-08-08. This is WHERE BLOCK REWARDS ARE PAID.
And unlike fulcrum and bitcoin-knots, the `Restart datum-gateway` handler here
was never gated, so the stale value would have applied immediately rather than
sitting inert on disk.
- pool_pass_workers / pool_pass_full_users: false on the node, true in the vars
file.
Both corrected in the vault and role defaults with notes recording why.
Comparing this config needs semantics, not text: the live file is single-line
JSON and the template renders pretty-printed, so a textual diff is pure noise.
Rendering it and comparing parsed JSON is what surfaced both differences.
config.json carries bitcoind.rpcpassword and api.admin_password, and --diff
prints rendered content - so `--check --diff` put them on the terminal. The task
now sets diff: false by default (-e datum_reveal_config=true to opt in). Those
two should be rotated.
I also mis-reported pool_pass_workers/pool_pass_full_users as exposed credentials
because my masking matched "pass" in the key name. They are BOOLEANS, and
mining.pool_address is a Bitcoin address, public by nature. Only the two real
passwords above were exposed.
`Configure cmake build` and `Compile datum_gateway` are bare command: tasks with
no changed_when, so they recompile on every run. The build is reproducible -
Install datum_gateway binary sees identical content and leaves the installed
binary's timestamp alone - but it is wasted work each time. Documented as the
idempotent floor.
Ownership parity checked mechanically against `git show HEAD:` keyed by task
name: 7/7 match, 9 Kuma tasks dropped.
This completes Plan 6 Stage 2: all six services in the list are roles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
