2025-11-06 23:09:44 +01:00
|
|
|
# Centralized Services Configuration
|
|
|
|
|
# Subdomains and Caddy settings for all services
|
|
|
|
|
|
|
|
|
|
# Edit these subdomains to match your preferences
|
|
|
|
|
subdomains:
|
|
|
|
|
# Monitoring Services (on watchtower)
|
2025-12-01 11:16:47 +01:00
|
|
|
ntfy: ntfy
|
2026-09-11 22:43:56 +02:00
|
|
|
# DEPRECATED 2026-09-11 — Uptime Kuma is decommissioned and this subdomain no
|
|
|
|
|
# longer resolves to anything. Kept only because the deprecated monitoring
|
|
|
|
|
# blocks still template it into uptime_kuma_api_url. See archive/uptime_kuma/.
|
2025-12-01 11:16:47 +01:00
|
|
|
uptime_kuma: uptime
|
2025-11-06 23:09:44 +01:00
|
|
|
|
|
|
|
|
# VPN Infrastructure (on spacey)
|
2025-12-01 11:16:47 +01:00
|
|
|
headscale: headscale
|
2025-11-06 23:09:44 +01:00
|
|
|
|
|
|
|
|
# Core Services (on vipy)
|
2025-12-01 11:16:47 +01:00
|
|
|
vaultwarden: vault
|
2025-12-01 12:14:25 +01:00
|
|
|
forgejo: forgejo
|
2025-12-06 23:44:17 +01:00
|
|
|
lnbits: wallet
|
2025-11-06 23:09:44 +01:00
|
|
|
|
|
|
|
|
# Secondary Services (on vipy)
|
2025-12-07 19:02:50 +01:00
|
|
|
ntfy_emergency_app: avisame
|
2025-12-06 23:44:17 +01:00
|
|
|
personal_blog: pablohere
|
2025-11-06 23:09:44 +01:00
|
|
|
|
|
|
|
|
# Memos (on memos-box)
|
2025-12-01 11:16:47 +01:00
|
|
|
memos: memos
|
2025-12-14 22:15:29 +01:00
|
|
|
|
|
|
|
|
# Mempool Block Explorer (on mempool_box, proxied via vipy)
|
|
|
|
|
mempool: mempool
|
2025-11-06 23:09:44 +01:00
|
|
|
|
2026-08-08 12:00:27 +02:00
|
|
|
# DATUM Gateway dashboard (on knots_box, proxied via vipy)
|
|
|
|
|
datum_gateway: datum
|
|
|
|
|
|
2025-11-06 23:09:44 +01:00
|
|
|
# Caddy configuration
|
|
|
|
|
caddy_sites_dir: /etc/caddy/sites-enabled
|
2025-11-14 23:36:00 +01:00
|
|
|
|
|
|
|
|
# Service-specific settings shared across playbooks
|
|
|
|
|
service_settings:
|
|
|
|
|
ntfy:
|
|
|
|
|
topic: alerts
|
|
|
|
|
headscale:
|
|
|
|
|
namespace: counter-net
|
mempool: convert to a role, de-Uptime-Kuma the health checks
745-line playbook becomes 37 lines (the role, plus the Caddy play for the edge
host) and a 408-line role with docker/deploy/healthcheck phases and six
templates. mempool_vars.yml is deleted; its content is the role's defaults.
Three health checks are kept, not collapsed: Mempool is three moving parts and
knowing which one is down is the point. Each has its own script, unit, timer and
push_url, driven by a mempool_healthchecks list. The Uptime Kuma specifics are
gone - the embedded Python creating monitors over the API, the /tmp credentials
file, the push-URL file read back and parsed, three Environment= rewrites - and
the three live push URLs are preserved from the vault, so reporting is unchanged.
`Enable and start health check timers` and `Display deployment status` were both
guarded by uptime_kuma_enabled despite being deployment tasks. Third service in
a row with that pattern: the deprecation banner was applied to contiguous blocks,
so anything sitting near the push plumbing was disabled with it. Ungated.
TWO OWNERSHIP PROBLEMS, different in kind:
- MINE: I wrote `owner: root` on docker-compose.yml where the original says
`owner: "{{ ansible_user }}"`. A straight violation of extract-mechanically-
change-nothing, caught only by reading the check-mode diff line by line.
Reverted to match the original.
- PRE-EXISTING, and dangerous: the playbook declared
`owner: "{{ ansible_user }}"` (1000) on the MariaDB data directory, which the
container owns as uid 999. Confirmed against `git show HEAD:` before
concluding it was not mine. It had drifted since the containers were created
and went unnoticed because the playbook had not been run since.
This was not academic. The first real run pulled a newer mariadb:10.11 and
recreated mempool-db; with the chown still in place MariaDB would have come
back to a data directory it could not write. The role now ensures the
directory exists and leaves ownership to the container. Verified after the
run: /opt/mempool/mysql is still 999:999 and all three containers are healthy.
This is a deliberate behaviour change, not part of the extraction. It is in
this commit rather than a follow-up because the faithful version was never
safe to run, so there was no intermediate state worth recording as verified.
mempool_frontend_port moved to services_config.yml: two hosts need it (this role
deploys the frontend, the Caddy play proxies to it from the edge host) and a role
default is invisible to the second play. caddy_site's parameter assert caught
this loudly - "'mempool_frontend_port' is undefined" - rather than silently.
Verified: check-mode diff clean apart from unavoidable check-mode artifacts;
first run ok=24 changed=5, zero failures; second run changed=2 - the two bare
`command:` tasks (pull, compose up) that have no changed_when and always report
changed. That is the idempotent floor. All three health checks report
ExecMainStatus 0 with their push URLs intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 18:49:11 +02:00
|
|
|
mempool:
|
|
|
|
|
# The frontend port is needed in two places on two different hosts: the
|
|
|
|
|
# mempool role deploys it on mempool-box, and the Caddy play proxies to it
|
|
|
|
|
# from the edge host. A role default cannot serve the second play, so it
|
|
|
|
|
# lives here rather than in roles/mempool/defaults.
|
|
|
|
|
frontend_port: 8080
|
bitcoin-knots: convert to a role, de-Uptime-Kuma the health check
892-line playbook becomes 40 lines plus a role with install/build/configure/
service/healthcheck phases, five templates and one handler. bitcoin_knots_vars.yml
is deleted; its content is the role's defaults.
Verified after a real run: bitcoind still active since 2026-08-19 (NO restart),
chain at 966844 blocks / 875 GB, DATUM config intact, dbcache still 200, health
check timer firing again. changed=3, all health-check. vipy changed=0.
⚠ THE BIG ONE: the playbook would have deleted the mining integration.
bitcoin.conf on the node carries a section that was hand-added and was missing
from the template entirely:
blockmaxsize=3985000
blockmaxweight=3985000
blocknotify=killall -USR1 datum_gateway
maxmempool=1000
blockreconstructionextratxn=1000000
blocknotify is how datum_gateway learns a new block landed. Running the old
playbook would have stripped all of it and solo mining would have carried on
against a stale template - a silent failure that costs money rather than raising
an error. Also dbcache 200 -> 3528 (hand-tuned down; the calculation wants 90% of
RAM) and logging moved off the file. All now reconciled, dbcache behind
bitcoin_dbcache_mb_override.
bitcoin-knots and datum-gateway are ONE SYSTEM. Noted in the README.
AND MY OWN FIX MADE IT MORE DANGEROUS. The `Restart bitcoind` handler was guarded
by uptime_kuma_enabled, so it had been inert: bitcoin.conf and the systemd unit
both notify it and neither could restart anything - a config change applied to
disk, reported success, and never took effect. Ungating that is right, but it
converts "wrong config sitting inertly on disk" into "node restarted onto a
config that breaks mining". The ungating had to land WITH the template
reconciliation, not before it.
It also raises the bar permanently: any residual template/live difference now
restarts a Bitcoin node on every run. Four rounds of --check --diff to reach
changed=0 - the DATUM section, an explanatory comment that was rendering into the
deployed config (now a {# #} Jinja comment), a "# Pruning (optional)" comment the
live file had, and one trailing blank line.
The build path is 32 tasks all guarded by `not bitcoind_binary_exists.stat.exists`,
so a converged host skips the 30-60 minute compile and both `state: absent`
deletions. Those target /opt/bitcoin-knots/{source,bitcoin-<version>}; the chain
is in /mnt/knots_data and is never touched. Signature-verification tasks copied
verbatim.
The health check timer had last fired 2026-08-09 while reporting active/enabled -
same OnBootSec + OnUnitActiveSec dead chain as fulcrum. The role runs the check
once after enabling to supply the reference the timer schedules from.
Ownership parity checked mechanically against `git show HEAD:`, keyed by TASK
NAME rather than path - keying by path gave a false positive, because
bitcoin_knots_source_dir is created with ownership and later removed with
state: absent, so whichever task comes last wins and that differs between one
file and five. 13/13 match.
bitcoin_p2p_port and the tailscale hostname moved to services_config.yml for the
socket-proxy play on the edge host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 18:13:40 +02:00
|
|
|
bitcoin:
|
|
|
|
|
# The P2P port is needed on two hosts: the bitcoin_knots role deploys the
|
|
|
|
|
# node on knots-box, and the socket-proxy play publishes the port from the
|
|
|
|
|
# edge host. A role default cannot reach that second play.
|
|
|
|
|
p2p_port: 8333
|
|
|
|
|
tailscale_hostname: knots-box
|
datum-gateway: convert to a role, de-Uptime-Kuma the health check
802-line playbook becomes 68 lines (three plays: the role, the Caddy dashboard,
the Stratum socket proxy) plus a 345-line role. datum_gateway_vars.yml is
deleted; its content is the role's defaults.
Verified after a real run with zero miners connected: datum-gateway restarted
cleanly onto the reformatted config, deployed config.json semantically identical
to what was there (pool_address bc1qvrj3g84..., pool_pass_* false, ports
unchanged), health check timer firing, and the Knots side untouched - bitcoind
still up since 2026-08-19 with blocknotify intact.
TWO PIECES OF DRIFT WHERE THE NODE WAS RIGHT, both confirmed with the operator:
- datum_mining_address: the vault held bc1qdse9dsg... while the node had been
mining to bc1qvrj3g... since 2026-08-08. This is WHERE BLOCK REWARDS ARE PAID.
And unlike fulcrum and bitcoin-knots, the `Restart datum-gateway` handler here
was never gated, so the stale value would have applied immediately rather than
sitting inert on disk.
- pool_pass_workers / pool_pass_full_users: false on the node, true in the vars
file.
Both corrected in the vault and role defaults with notes recording why.
Comparing this config needs semantics, not text: the live file is single-line
JSON and the template renders pretty-printed, so a textual diff is pure noise.
Rendering it and comparing parsed JSON is what surfaced both differences.
config.json carries bitcoind.rpcpassword and api.admin_password, and --diff
prints rendered content - so `--check --diff` put them on the terminal. The task
now sets diff: false by default (-e datum_reveal_config=true to opt in). Those
two should be rotated.
I also mis-reported pool_pass_workers/pool_pass_full_users as exposed credentials
because my masking matched "pass" in the key name. They are BOOLEANS, and
mining.pool_address is a Bitcoin address, public by nature. Only the two real
passwords above were exposed.
`Configure cmake build` and `Compile datum_gateway` are bare command: tasks with
no changed_when, so they recompile on every run. The build is reproducible -
Install datum_gateway binary sees identical content and leaves the installed
binary's timestamp alone - but it is wasted work each time. Documented as the
idempotent floor.
Ownership parity checked mechanically against `git show HEAD:` keyed by task
name: 7/7 match, 9 Kuma tasks dropped.
This completes Plan 6 Stage 2: all six services in the list are roles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 18:25:41 +02:00
|
|
|
datum_gateway:
|
|
|
|
|
# Needed on two hosts: the datum_gateway role deploys on knots-box, while the
|
|
|
|
|
# Caddy play (dashboard) and the socket-proxy play (Stratum) both run on the
|
|
|
|
|
# edge host. A role default cannot reach either of those plays.
|
|
|
|
|
api_port: 7152
|
|
|
|
|
stratum_port: 23334
|
|
|
|
|
tailscale_hostname: knots-box
|
fulcrum: convert to a role, de-Uptime-Kuma the health check
685-line playbook becomes 33 lines plus a 426-line role
(install/service/healthcheck phases, six templates, one handler).
fulcrum_vars.yml is deleted; its content is the role's defaults.
Verified: fulcrum untouched - active since 2026-07-29 (no restart), 192G datadir,
height 966842, bitcoind and db_mem unchanged on disk. Second run changed=1 (the
arming run, changed_when: false aside). vipy changed=0.
THREE PRE-EXISTING LANDMINES the check-mode diff caught, any of which a faithful
extraction would have detonated:
- bitcoin_rpc_host was "192.168.1.140", commented "IP of knots_box_local". But
.140 is fulcrum-box ITSELF; knots-box is .135. The DHCP leases had reshuffled -
the fifth instance of this same disease in this estate. The live config had
been hand-corrected to knots-box; running the playbook would have reverted it
and pointed Fulcrum at itself. Now addressed by Tailscale name.
- The `Restart fulcrum` handler was guarded by uptime_kuma_enabled, so the three
tasks that notify it (SSL cert, fulcrum.conf, systemd unit) could not restart
anything. A config change applied to disk, reported success, and silently never
took effect. That is worse than the other banner casualties: it makes the
deployment itself lie. Ungated.
- db_mem was about to go 2048 -> 4448 (75% of 5931MB RAM), leaving ~1.4GB for the
OS and Fulcrum's non-cache memory. The live value had been hand-tuned down.
fulcrum_db_mem_mb_override pins it. Note set_fact outranks role defaults, so
the calculation itself has to honour the override.
MY OWN ERROR, third instance: retyping `copy:` as `template:` lost `owner:` on
the banner and on fulcrum.conf. Rather than keep catching these by eye, every
managed path's owner/group/mode is now compared against `git show HEAD:`
mechanically - 12/12 match.
The health check timer had not fired since 2026-02-17 while reporting `active`
and `enabled`. It is OnBootSec + OnUnitActiveSec with no OnCalendar: OnBootSec
elapses once, and OnUnitActiveSec needs the SERVICE to have run this boot to have
anything to schedule from. Restarting the timer does not supply that; running the
service does, so the role now runs the check once after enabling. Also dropped
`Requires=fulcrum.service` from the timer - on a timer that means "stop watching
when the watched thing stops".
Diagnostic note: NextElapseUSecRealtime is always empty for a monotonic timer, so
it reads as broken even when healthy. I misread it once and wrongly called the
timer dead. Use NextElapseUSecMonotonic or systemctl list-timers.
fulcrum_ssl_port and fulcrum_tailscale_hostname moved to services_config.yml -
the socket-proxy play on the edge host needs them and a role default cannot reach
a second play.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 18:02:00 +02:00
|
|
|
fulcrum:
|
|
|
|
|
# Same shape as mempool: the fulcrum role deploys on fulcrum-box, and the
|
|
|
|
|
# socket-proxy play publishes the SSL port from the edge host. A role default
|
|
|
|
|
# is invisible to that second play.
|
|
|
|
|
ssl_port: 50002
|
|
|
|
|
tailscale_hostname: fulcrum-box
|