685-line playbook becomes 33 lines plus a 426-line role
(install/service/healthcheck phases, six templates, one handler).
fulcrum_vars.yml is deleted; its content is the role's defaults.
Verified: fulcrum untouched - active since 2026-07-29 (no restart), 192G datadir,
height 966842, bitcoind and db_mem unchanged on disk. Second run changed=1 (the
arming run, changed_when: false aside). vipy changed=0.
THREE PRE-EXISTING LANDMINES the check-mode diff caught, any of which a faithful
extraction would have detonated:
- bitcoin_rpc_host was "192.168.1.140", commented "IP of knots_box_local". But
.140 is fulcrum-box ITSELF; knots-box is .135. The DHCP leases had reshuffled -
the fifth instance of this same disease in this estate. The live config had
been hand-corrected to knots-box; running the playbook would have reverted it
and pointed Fulcrum at itself. Now addressed by Tailscale name.
- The `Restart fulcrum` handler was guarded by uptime_kuma_enabled, so the three
tasks that notify it (SSL cert, fulcrum.conf, systemd unit) could not restart
anything. A config change applied to disk, reported success, and silently never
took effect. That is worse than the other banner casualties: it makes the
deployment itself lie. Ungated.
- db_mem was about to go 2048 -> 4448 (75% of 5931MB RAM), leaving ~1.4GB for the
OS and Fulcrum's non-cache memory. The live value had been hand-tuned down.
fulcrum_db_mem_mb_override pins it. Note set_fact outranks role defaults, so
the calculation itself has to honour the override.
MY OWN ERROR, third instance: retyping `copy:` as `template:` lost `owner:` on
the banner and on fulcrum.conf. Rather than keep catching these by eye, every
managed path's owner/group/mode is now compared against `git show HEAD:`
mechanically - 12/12 match.
The health check timer had not fired since 2026-02-17 while reporting `active`
and `enabled`. It is OnBootSec + OnUnitActiveSec with no OnCalendar: OnBootSec
elapses once, and OnUnitActiveSec needs the SERVICE to have run this boot to have
anything to schedule from. Restarting the timer does not supply that; running the
service does, so the role now runs the check once after enabling. Also dropped
`Requires=fulcrum.service` from the timer - on a timer that means "stop watching
when the watched thing stops".
Diagnostic note: NextElapseUSecRealtime is always empty for a monotonic timer, so
it reads as broken even when healthy. I misread it once and wrongly called the
timer dead. Use NextElapseUSecMonotonic or systemctl list-timers.
fulcrum_ssl_port and fulcrum_tailscale_hostname moved to services_config.yml -
the socket-proxy play on the edge host needs them and a role default cannot reach
a second play.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All five close-out greps return nothing: no sites-enabled handling outside
roles/, no `systemctl reload caddy`, no caddy_sites_dir self-reference, no
inline proxy unit writes. 37 playbooks syntax clean. The 14 Caddy site files
and 6 proxy units on the hosts are byte-identical to the Stage 0 baseline.
Seven play names still said "on vipy" while the play targeted a group. Renamed
to "on the edge host" - the last place a play claimed a hostname after Plan 2.
Documented the four vhosts in /etc/caddy/sites-enabled that no playbook writes
(uptime-kuma, arbretstaging, bitcoininfra, scriberr) in the caddy_site README.
None deleted.
uptime-kuma.conf was going to be deleted as dead config. It is not dead: the
louislam/uptime-kuma container is STILL RUNNING on watchtower - created
2026-02-07, restart=unless-stopped, healthy - and uptime.contrapeso.xyz returns
302, not the 502 a dead backend would give. The "decommissioning" retired the
Ansible code and the vault credentials, not the service. PLAN_3 claimed "the
tokens died with the server"; that is corrected there.
The Caddyfile.* backups are kept: one per host, Nov-Dec 2025, not churning, and
the only record of each Caddyfile before the import line was added.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three near-identical hosts: edge plays become one role plus three short
calls. 183 lines removed, 34 added, plus a 111-line role.
Verified before touching any playbook: all six live units on vipy reproduced
byte-identically. Then --limit edge --check per playbook - bitcoin-knots and
fulcrum changed=0; datum-gateway changed=2, both attributable to the already
known caddy_site comment line and the Reload caddy handler it triggers.
The 6 units and 14 Caddy files on the hosts are byte-identical afterwards.
PLAN_4 claimed these three plays had "no behavioural drift at all". That was
wrong - it came from a diff truncated by head -60. The live bitcoin-p2p-proxy
units carry four settings this playbook never wrote:
.socket Documentation=, FreeBind=true
.service Documentation=, TimeoutStopSec=5,
StandardOutput=journal, StandardError=journal
FreeBind is the one that matters: it lets the socket bind to an address that
is not up yet, so without it the socket can fail to start on boot. Running
the bitcoin-knots playbook would have stripped it. Same class of hazard as
headscale. The role expresses all four; bitcoin-p2p is the only caller that
passes any.
Also: UFW treats the rule comment as part of the rule. datum-stratum's live
comment is "DATUM Gateway Stratum public access" but the role's derived
default produced "DATUM Stratum public access", which rewrote the rule.
Caught in the dry-run; datum now passes the comment explicitly.
Two deliberate differences from the original, both documented in the README:
ignore_errors: yes on the upstream check became failed_when: false, and the
handler restarts the .socket, which drops connections open through it - it
fires only when a unit file actually changes.
The inert Uptime Kuma TCP monitor blocks stay in the playbooks rather than
being pulled into a new role (12/12/18 guarded tasks).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the credentials would otherwise break these playbooks mid-deploy: they
template uptime_kuma_password with no assert to stop them first. 100 tasks are
now guarded by uptime_kuma_enabled (false), so deployments run normally and the
monitoring sections skip. A further 28 tasks were already self-guarding on
monitor_setup/push_url being defined; verified that a skipped task's registered
variable makes those skip cleanly rather than error.
The blocks are kept on purpose — the health-check logic is the durable part and
should be rewired to whatever replaces Uptime Kuma.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
