bitcoin-knots: ride out block-time RPC stalls

The RPC probe flapped while every other knots check stayed green. Each burst
lined up with a block: validation on the saturated HDD holds the node busy
while Fulcrum, mempool and DATUM all fetch the block, and bitcoind rejected
calls with "work queue depth exceeded" (93 times on 2026-10-18).

- 403: the probe runs every minute, so failure-threshold 4 rides out a
  1-3 minute burst while a real outage still alerts within ~5 minutes.
  Probes now take an optional failure_threshold over a shared probe_alert.
- rpcworkqueue=64 and rpcthreads=8, so bursts queue instead of failing.

Profiling later showed the stalls are per-input UTXO lookups at ~5 ms each -
HDD seek latency - so this treats the symptom. The fix is chainstate on SSD.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ansible/roles/bitcoin_knots/defaults/main.yml

@@ -22,6 +22,15 @@ bitcoin_rpc_port: 8332
 bitcoin_p2p_port: 8333
 bitcoin_rpc_bind: "0.0.0.0"
 
+# RPC capacity. When a block lands, Fulcrum, mempool and DATUM all fetch it at
+# once while validation holds the node busy on a saturated HDD; with the
+# defaults the queue overflowed and bitcoind REJECTED calls ("http work queue
+# depth exceeded" - 93 times on 2026-10-18), which also made the health probe
+# flap. A deeper queue lets those bursts wait instead of failing. Costs a few
+# idle threads and a little memory.
+bitcoin_rpcworkqueue: 64
+bitcoin_rpcthreads: 8
+
 # Build options
 bitcoin_build_jobs: 4  # Parallel build jobs (-j flag), adjust based on CPU cores
 bitcoin_build_prefix: /usr/local