personal_infra/ansible/roles/bitcoin_knots
counterweight 2a80f0e3ca
bitcoin-knots: ride out block-time RPC stalls
The RPC probe flapped while every other knots check stayed green. Each burst
lined up with a block: validation on the saturated HDD holds the node busy
while Fulcrum, mempool and DATUM all fetch the block, and bitcoind rejected
calls with "work queue depth exceeded" (93 times on 2026-10-18).

- 403: the probe runs every minute, so failure-threshold 4 rides out a
  1-3 minute burst while a real outage still alerts within ~5 minutes.
  Probes now take an optional failure_threshold over a shared probe_alert.
- rpcworkqueue=64 and rpcthreads=8, so bursts queue instead of failing.

Profiling later showed the stalls are per-input UTXO lookups at ~5 ms each -
HDD seek latency - so this treats the symptom. The fix is chainstate on SSD.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 22:10:25 +02:00
..
defaults bitcoin-knots: ride out block-time RPC stalls 2026-10-19 22:10:25 +02:00
handlers bitcoin-knots: convert to a role, de-Uptime-Kuma the health check 2026-09-13 18:13:40 +02:00
tasks uptime kuma: remove every live reference, repoint the probes to Gatus 2026-09-14 10:30:28 +02:00
templates bitcoin-knots: ride out block-time RPC stalls 2026-10-19 22:10:25 +02:00
README.md bitcoin-knots: convert to a role, de-Uptime-Kuma the health check 2026-09-13 18:13:40 +02:00

bitcoin_knots

Builds Bitcoin Knots from source with PGP + SHA256 verification of the release tarball, runs it as a full node on knots-box, and keeps a health check on a systemd timer. The second play in the calling playbook publishes the P2P port from the edge host via socket_proxy.

Converted from deploy_bitcoin_knots_playbook.yml (892 lines) under Plan 6. The playbook is now 40 lines.

The build is guarded; the chain is never touched

build.yml is 32 tasks, every one carrying when: not bitcoind_binary_exists.stat.exists. On a host that already has the binary the whole download / verify / 30-60 minute compile skips — including the two state: absent deletions, which target /opt/bitcoin-knots/source and the extracted build directory.

The chain lives elsewhere and nothing here touches it:

bitcoin_knots_dir /opt/bitcoin-knots — build tree, safe to delete
bitcoin_data_dir /var/lib/bitcoin — config, logs, wallets
bitcoin_large_data_dir /mnt/knots_data — ~875 GB of blockchain

The signature-verification tasks are the security control of this role. They are copied verbatim; do not "simplify" them.

⚠ This node is half of the mining setup

bitcoin.conf carries a DATUM Gateway section that was hand-added on the node and was missing from the playbook's template:

blockmaxsize=3985000
blockmaxweight=3985000
blocknotify=killall -USR1 datum_gateway
maxmempool=1000
blockreconstructionextratxn=1000000

blocknotify is how datum_gateway learns a new block landed. Running the old playbook would have deleted all of it, and solo mining would have carried on grinding against a stale template — a silent failure that costs money rather than raising an error. The template now carries it behind bitcoin_datum_gateway_enabled.

bitcoin-knots and datum-gateway are one system, not two services. Changing either config means thinking about both.

The restart handler, and why exactness matters now

The hand-written Restart bitcoind handler carried when: uptime_kuma_enabled | default(false), so it had been inert since the decommissioning: bitcoin.conf and the systemd unit both notify it and neither could restart anything. A config change applied to disk, reported success, and never took effect.

It is ungated here — which raises the bar for the template. Any residual difference between the template and the live file, down to a trailing newline, means the task reports changed and restarts a Bitcoin node on every run. It took four rounds of --check --diff to reach changed=0: the DATUM section, an explanatory comment that was rendering into the deployed file (now a {# #} Jinja comment), a # Pruning (optional) comment the live file had, and one trailing blank line.

dbcache

Computed as 90% of RAM unless bitcoin_dbcache_mb_override is set. The live node was hand-tuned to 200 MB; the calculation produces 3528. As with fulcrum, set_fact outranks role defaults, so the calculation honours the override — a value pinned only in defaults/ is silently ignored.

Monitoring: one variable, no product knowledge

The check tests bitcoind's RPC and records the answer in its exit code, which systemd keeps: systemctl is-failed bitcoin-knots-healthcheck.service. Set healthcheck_push_url to report anywhere that accepts an HTTP ping.

The timer had last fired 2026-08-09 while still reporting active and enabled — the same OnBootSec + OnUnitActiveSec dead chain as fulcrum, where nothing re-arms it if the service does not run in a given boot. The role runs the check once after enabling, which both smoke-tests it and supplies the reference the timer schedules from.