The RPC probe flapped while every other knots check stayed green. Each burst lined up with a block: validation on the saturated HDD holds the node busy while Fulcrum, mempool and DATUM all fetch the block, and bitcoind rejected calls with "work queue depth exceeded" (93 times on 2026-10-18). - 403: the probe runs every minute, so failure-threshold 4 rides out a 1-3 minute burst while a real outage still alerts within ~5 minutes. Probes now take an optional failure_threshold over a shared probe_alert. - rpcworkqueue=64 and rpcthreads=8, so bursts queue instead of failing. Profiling later showed the stalls are per-input UTXO lookups at ~5 ms each - HDD seek latency - so this treats the symptom. The fix is chainstate on SSD. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| defaults | ||
| handlers | ||
| tasks | ||
| templates | ||
| README.md | ||
bitcoin_knots
Builds Bitcoin Knots from source with PGP + SHA256 verification of the release
tarball, runs it as a full node on knots-box, and keeps a health check on a
systemd timer. The second play in the calling playbook publishes the P2P port
from the edge host via socket_proxy.
Converted from deploy_bitcoin_knots_playbook.yml (892 lines) under Plan 6. The
playbook is now 40 lines.
The build is guarded; the chain is never touched
build.yml is 32 tasks, every one carrying
when: not bitcoind_binary_exists.stat.exists. On a host that already has the
binary the whole download / verify / 30-60 minute compile skips — including the
two state: absent deletions, which target /opt/bitcoin-knots/source and the
extracted build directory.
The chain lives elsewhere and nothing here touches it:
bitcoin_knots_dir |
/opt/bitcoin-knots — build tree, safe to delete |
bitcoin_data_dir |
/var/lib/bitcoin — config, logs, wallets |
bitcoin_large_data_dir |
/mnt/knots_data — ~875 GB of blockchain |
The signature-verification tasks are the security control of this role. They are copied verbatim; do not "simplify" them.
⚠ This node is half of the mining setup
bitcoin.conf carries a DATUM Gateway section that was hand-added on the node
and was missing from the playbook's template:
blockmaxsize=3985000
blockmaxweight=3985000
blocknotify=killall -USR1 datum_gateway
maxmempool=1000
blockreconstructionextratxn=1000000
blocknotify is how datum_gateway learns a new block landed. Running the old
playbook would have deleted all of it, and solo mining would have carried on
grinding against a stale template — a silent failure that costs money rather
than raising an error. The template now carries it behind
bitcoin_datum_gateway_enabled.
bitcoin-knots and datum-gateway are one system, not two services. Changing either config means thinking about both.
The restart handler, and why exactness matters now
The hand-written Restart bitcoind handler carried
when: uptime_kuma_enabled | default(false), so it had been inert since the
decommissioning: bitcoin.conf and the systemd unit both notify it and neither
could restart anything. A config change applied to disk, reported success, and
never took effect.
It is ungated here — which raises the bar for the template. Any residual
difference between the template and the live file, down to a trailing newline,
means the task reports changed and restarts a Bitcoin node on every run. It
took four rounds of --check --diff to reach changed=0: the DATUM section, an
explanatory comment that was rendering into the deployed file (now a {# #}
Jinja comment), a # Pruning (optional) comment the live file had, and one
trailing blank line.
dbcache
Computed as 90% of RAM unless bitcoin_dbcache_mb_override is set. The live node
was hand-tuned to 200 MB; the calculation produces 3528. As with fulcrum,
set_fact outranks role defaults, so the calculation honours the override — a
value pinned only in defaults/ is silently ignored.
Monitoring: one variable, no product knowledge
The check tests bitcoind's RPC and records the answer in its exit code, which
systemd keeps: systemctl is-failed bitcoin-knots-healthcheck.service. Set
healthcheck_push_url to report anywhere that accepts an HTTP ping.
The timer had last fired 2026-08-09 while still reporting active and
enabled — the same OnBootSec + OnUnitActiveSec dead chain as fulcrum, where
nothing re-arms it if the service does not run in a given boot. The role runs the
check once after enabling, which both smoke-tests it and supplies the reference
the timer schedules from.