685-line playbook becomes 33 lines plus a 426-line role
(install/service/healthcheck phases, six templates, one handler).
fulcrum_vars.yml is deleted; its content is the role's defaults.
Verified: fulcrum untouched - active since 2026-07-29 (no restart), 192G datadir,
height 966842, bitcoind and db_mem unchanged on disk. Second run changed=1 (the
arming run, changed_when: false aside). vipy changed=0.
THREE PRE-EXISTING LANDMINES the check-mode diff caught, any of which a faithful
extraction would have detonated:
- bitcoin_rpc_host was "192.168.1.140", commented "IP of knots_box_local". But
.140 is fulcrum-box ITSELF; knots-box is .135. The DHCP leases had reshuffled -
the fifth instance of this same disease in this estate. The live config had
been hand-corrected to knots-box; running the playbook would have reverted it
and pointed Fulcrum at itself. Now addressed by Tailscale name.
- The `Restart fulcrum` handler was guarded by uptime_kuma_enabled, so the three
tasks that notify it (SSL cert, fulcrum.conf, systemd unit) could not restart
anything. A config change applied to disk, reported success, and silently never
took effect. That is worse than the other banner casualties: it makes the
deployment itself lie. Ungated.
- db_mem was about to go 2048 -> 4448 (75% of 5931MB RAM), leaving ~1.4GB for the
OS and Fulcrum's non-cache memory. The live value had been hand-tuned down.
fulcrum_db_mem_mb_override pins it. Note set_fact outranks role defaults, so
the calculation itself has to honour the override.
MY OWN ERROR, third instance: retyping `copy:` as `template:` lost `owner:` on
the banner and on fulcrum.conf. Rather than keep catching these by eye, every
managed path's owner/group/mode is now compared against `git show HEAD:`
mechanically - 12/12 match.
The health check timer had not fired since 2026-02-17 while reporting `active`
and `enabled`. It is OnBootSec + OnUnitActiveSec with no OnCalendar: OnBootSec
elapses once, and OnUnitActiveSec needs the SERVICE to have run this boot to have
anything to schedule from. Restarting the timer does not supply that; running the
service does, so the role now runs the check once after enabling. Also dropped
`Requires=fulcrum.service` from the timer - on a timer that means "stop watching
when the watched thing stops".
Diagnostic note: NextElapseUSecRealtime is always empty for a monotonic timer, so
it reads as broken even when healthy. I misread it once and wrongly called the
timer dead. Use NextElapseUSecMonotonic or systemctl list-timers.
fulcrum_ssl_port and fulcrum_tailscale_hostname moved to services_config.yml -
the socket-proxy play on the edge host needs them and a role default cannot reach
a second play.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
745-line playbook becomes 37 lines (the role, plus the Caddy play for the edge
host) and a 408-line role with docker/deploy/healthcheck phases and six
templates. mempool_vars.yml is deleted; its content is the role's defaults.
Three health checks are kept, not collapsed: Mempool is three moving parts and
knowing which one is down is the point. Each has its own script, unit, timer and
push_url, driven by a mempool_healthchecks list. The Uptime Kuma specifics are
gone - the embedded Python creating monitors over the API, the /tmp credentials
file, the push-URL file read back and parsed, three Environment= rewrites - and
the three live push URLs are preserved from the vault, so reporting is unchanged.
`Enable and start health check timers` and `Display deployment status` were both
guarded by uptime_kuma_enabled despite being deployment tasks. Third service in
a row with that pattern: the deprecation banner was applied to contiguous blocks,
so anything sitting near the push plumbing was disabled with it. Ungated.
TWO OWNERSHIP PROBLEMS, different in kind:
- MINE: I wrote `owner: root` on docker-compose.yml where the original says
`owner: "{{ ansible_user }}"`. A straight violation of extract-mechanically-
change-nothing, caught only by reading the check-mode diff line by line.
Reverted to match the original.
- PRE-EXISTING, and dangerous: the playbook declared
`owner: "{{ ansible_user }}"` (1000) on the MariaDB data directory, which the
container owns as uid 999. Confirmed against `git show HEAD:` before
concluding it was not mine. It had drifted since the containers were created
and went unnoticed because the playbook had not been run since.
This was not academic. The first real run pulled a newer mariadb:10.11 and
recreated mempool-db; with the chown still in place MariaDB would have come
back to a data directory it could not write. The role now ensures the
directory exists and leaves ownership to the container. Verified after the
run: /opt/mempool/mysql is still 999:999 and all three containers are healthy.
This is a deliberate behaviour change, not part of the extraction. It is in
this commit rather than a follow-up because the faithful version was never
safe to run, so there was no intermediate state worth recording as verified.
mempool_frontend_port moved to services_config.yml: two hosts need it (this role
deploys the frontend, the Caddy play proxies to it from the edge host) and a role
default is invisible to the second play. caddy_site's parameter assert caught
this loudly - "'mempool_frontend_port' is undefined" - rather than silently.
Verified: check-mode diff clean apart from unavoidable check-mode artifacts;
first run ok=24 changed=5, zero failures; second run changed=2 - the two bare
`command:` tasks (pull, compose up) that have no changed_when and always report
changed. That is the idempotent floor. All three health checks report
ExecMainStatus 0 with their push URLs intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
552-line playbook becomes 18 lines plus a 411-line role
(install/service/healthcheck phases, four templates, two handlers).
phoenixd_vars.yml is deleted; its content is the role's defaults.
Task-list diff vs the old playbook shows ONLY the eight Uptime Kuma tasks
removed - everything else identical and in the same order.
phoenixd holds a Lightning node, so the run was checked against a pre-flight:
before: channel 6c25fa83..., balanceSat 1550723, capacitySat 3114830,
blockHeight 966692, active since 2026-09-02
after: identical, and still active since 2026-09-02 - it did NOT restart
`Create phoenixd systemd service` came back unchanged, which is what proves the
template reproduces the live unit byte-for-byte. changed=3 was the health check
script, its unit (Environment rename), and the timer restart. Second run:
changed=0.
Two things the conversion fixed, both symptoms of the deprecation banner having
been applied to contiguous blocks rather than to individual tasks:
- The health check logged "ERROR: UPTIME_KUMA_PUSH_URL not set" on every fire -
about 1,400 times a day - because its Environment= was emptied at
decommissioning. The exit code was still correct so nothing was broken, but it
is exactly the kind of noise that trains you to ignore a log. An unset push
URL is now normal and silent.
- `Enable and start phoenixd health check timer` was guarded by
uptime_kuma_enabled and so had not run since the decommissioning, while the
timer itself was still live on the host from before. Ansible had quietly
stopped managing something that was still running. Ungated.
Noted, not changed: seed.dat is mode 0644 on the host. That is phoenixd's own
doing, but it is a Lightning seed and worth tightening.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
409-line playbook becomes a 16-line playbook plus a 318-line role with phases
split across tasks/{prerequisites,install,configure,service,healthcheck}.yml and
four templates. forgejo_runner_vars.yml is deleted; its content is the role's
defaults.
Applies the Plan 6 Stage 0 decision: keep whatever determines whether the
service is healthy, drop the Uptime Kuma specifics, make the reporting point
pluggable. Gone from the role: the embedded Python that created monitors over
the Kuma API, the /tmp credentials file, token extraction, the systemd
Environment= rewrite, and 8 `when: uptime_kuma_enabled` guards. What remains is
the check itself, its log, the systemd unit and timer, and an honest exit code -
`systemctl is-failed forgejo-runner-healthcheck.service` now answers the
question with no monitoring system involved at all.
Reporting is one variable, healthcheck_push_url, empty by default. Any endpoint
that accepts an HTTP ping plugs in there. A pull-based monitor wants it left
empty and reads unit state instead.
PREMISE CORRECTION: Uptime Kuma is NOT dead. Plan 3 recorded "48 push timers
curling an endpoint that no longer answers" and Plan 6 said the check had
"nowhere to report to". Both wrong - 24+ push scripts across 11 hosts are
pushing successfully right now (HTTP 200). Only the Ansible code and the vault
credentials were decommissioned; the service never stopped. So the existing push
URLs were harvested into a vaulted healthcheck_push_urls dict and are preserved,
keeping this refactor behaviour-neutral. Retiring Kuma stays a deliberate act
rather than a side effect. PLAN_3 and PLAN_6 are corrected.
Verified:
- task-list diff vs the old playbook shows ONLY the five Kuma tasks removed,
everything else identical and in the same order
- first run ok=22 changed=1 (the rewritten health script); both systemd units
and forgejo-runner.service came back ok, so the templates reproduce the
previous files byte-for-byte
- second run ok=22 changed=0, fully idempotent
- still reports "Ping sent successfully (HTTP 200)" from a script containing
zero Uptime Kuma references
- the 4 skipped tasks are genuine already-configured guards, checked not assumed
Two things for the next service:
- import_tasks, not include_tasks. Dynamic includes are opaque to --list-tasks,
which is the primary verification tool here; the first attempt produced a
useless diff.
- `Assert runner is running` was guarded by uptime_kuma_enabled and so had not
run since the decommissioning. It is not monitoring, it is the deployment
checking its own work - the deprecation banner swept it up with the Kuma
plumbing, and a runner that failed to start was deploying "successfully" in
silence. Ungated now. The banner was applied to contiguous blocks, so read
every uptime_kuma_enabled guard and ask whether it is monitoring or deployment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All five close-out greps return nothing: no sites-enabled handling outside
roles/, no `systemctl reload caddy`, no caddy_sites_dir self-reference, no
inline proxy unit writes. 37 playbooks syntax clean. The 14 Caddy site files
and 6 proxy units on the hosts are byte-identical to the Stage 0 baseline.
Seven play names still said "on vipy" while the play targeted a group. Renamed
to "on the edge host" - the last place a play claimed a hostname after Plan 2.
Documented the four vhosts in /etc/caddy/sites-enabled that no playbook writes
(uptime-kuma, arbretstaging, bitcoininfra, scriberr) in the caddy_site README.
None deleted.
uptime-kuma.conf was going to be deleted as dead config. It is not dead: the
louislam/uptime-kuma container is STILL RUNNING on watchtower - created
2026-02-07, restart=unless-stopped, healthy - and uptime.contrapeso.xyz returns
302, not the 502 a dead backend would give. The "decommissioning" retired the
Ansible code and the vault credentials, not the service. PLAN_3 claimed "the
tokens died with the server"; that is corrected there.
The Caddyfile.* backups are kept: one per host, Nov-Dec 2025, not churning, and
the only record of each Caddyfile before the import line was added.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three near-identical hosts: edge plays become one role plus three short
calls. 183 lines removed, 34 added, plus a 111-line role.
Verified before touching any playbook: all six live units on vipy reproduced
byte-identically. Then --limit edge --check per playbook - bitcoin-knots and
fulcrum changed=0; datum-gateway changed=2, both attributable to the already
known caddy_site comment line and the Reload caddy handler it triggers.
The 6 units and 14 Caddy files on the hosts are byte-identical afterwards.
PLAN_4 claimed these three plays had "no behavioural drift at all". That was
wrong - it came from a diff truncated by head -60. The live bitcoin-p2p-proxy
units carry four settings this playbook never wrote:
.socket Documentation=, FreeBind=true
.service Documentation=, TimeoutStopSec=5,
StandardOutput=journal, StandardError=journal
FreeBind is the one that matters: it lets the socket bind to an address that
is not up yet, so without it the socket can fail to start on boot. Running
the bitcoin-knots playbook would have stripped it. Same class of hazard as
headscale. The role expresses all four; bitcoin-p2p is the only caller that
passes any.
Also: UFW treats the rule comment as part of the rule. datum-stratum's live
comment is "DATUM Gateway Stratum public access" but the role's derived
default produced "DATUM Stratum public access", which rewrote the rule.
Caught in the dry-run; datum now passes the comment explicitly.
Two deliberate differences from the original, both documented in the README:
ignore_errors: yes on the upstream check became failed_when: false, and the
handler restarts the .socket, which drops connections open through it - it
fires only when a unit file actually changes.
The inert Uptime Kuma TCP monitor blocks stay in the playbooks rather than
being pulled into a new role (12/12/18 guarded tasks).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes Stage 3. No hand-rolled Caddy plumbing remains anywhere:
`grep sites-enabled` outside roles/ returns nothing, and so does
`grep "systemctl reload caddy"`.
ntfy uses caddy_site_body for its plain-HTTP listener and @httpget redirect.
Verified ok/unchanged against watchtower; the one other changed task is a
pre-existing "Update APT cache".
datum-gateway keeps a whole-Caddyfile validate after the role call. The role
validates its own fragment, but only a whole-file validate catches a conflict
between two sites, and this playbook was the only one that ever had it. Its
two debug tasks that echoed command output are gone with the commands.
headscale is the one that mattered. Its playbook wrote
`reverse_proxy localhost:8080`, but spacey is actually running a /admin*
route in front of Headplane behind Caddy basic auth. Running that playbook
would have deleted the admin route and its auth - a hazard that predates this
work. It now renders the config that is really there, verified ok/unchanged
via --start-at-task (the play cannot reach Caddy in check mode: "Install
headscale package" fails because the .deb is not really downloaded, before
and after this edit alike).
Supporting changes for headscale:
- headscale_ui_password_hash added to infra_secrets.yml and the identical
group_vars/all/vault.yml, read from the live config on spacey. The vault
already had headscale_ui_username (= counterweight, confirmed) and
headscale_ui_password; I did not verify the password is the plaintext of
this hash.
- headplane_port added to headscale_vars.yml.
- The role's handler now sets become: true. Handlers do not inherit become
from the task that notified them, and this play runs become: no.
- The include uses `apply: become: yes`; `become:` on an include_role is
rejected outright.
All 14 site files on all 3 hosts still byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the four-task Caddy vhost block currently copy-pasted into 10
playbooks. Nothing calls it yet; this commit only adds the role.
Verified by rendering all 10 sites through the template and diffing against
what the current playbooks produce: 9 of 10 byte-identical. The tenth is
datum-gateway, where the resolvers comment is standardised, rewriting one
comment line Caddy ignores.
Then dry-run against the live hosts (--check, nothing written):
- vipy: forgejo, vaultwarden, lnbits, personal-blog, ntfy-emergency-app
all report ok/unchanged against the real files
- watchtower: ntfy renders identical via caddy_site_body, blank line and
{host}{uri} placeholders intact
- spacey: headscale renders identical when given the config that is
actually running
- memos, mempool, datum-gateway report changed - the comment, as expected
All 14 site files on all 3 hosts confirmed unchanged afterwards.
Two things the build turned up:
- Ansible does not template dict *keys*, so caddy_site_basic_auth is a list
of {user, hash}. As a dict, a Jinja username passes through literally.
The assert refuses a mapping.
- `caddy validate` does accept a single site fragment - rc=0 on a good one,
rc=1 with a line number on a broken one. This was the plan's one untested
claim. A failed validate leaves the live file untouched.
The reload is now a handler, so it fires once at end of play rather than
immediately; anything needing the new config live mid-play must
flush_handlers first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
