bitcoin-knots: ride out block-time RPC stalls

The RPC probe flapped while every other knots check stayed green. Each burst
lined up with a block: validation on the saturated HDD holds the node busy
while Fulcrum, mempool and DATUM all fetch the block, and bitcoind rejected
calls with "work queue depth exceeded" (93 times on 2026-10-18).

- 403: the probe runs every minute, so failure-threshold 4 rides out a
  1-3 minute burst while a real outage still alerts within ~5 minutes.
  Probes now take an optional failure_threshold over a shared probe_alert.
- rpcworkqueue=64 and rpcthreads=8, so bursts queue instead of failing.

Profiling later showed the stalls are per-input UTXO lookups at ~5 ms each -
HDD seek latency - so this treats the symptom. The fix is chainstate on SSD.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ansible/infra/403_service_probe_registration.yml

@@ -37,7 +37,13 @@
 
   vars:
     probes:
-      - {name: bitcoin-knots,    host: knots_box_local}
+      # failure_threshold 4: this probe runs every MINUTE, and bitcoind turns RPC
+      # away for 1-3 minutes when a block lands - validation on the saturated
+      # HDD plus Fulcrum/mempool/DATUM all fetching the block at once (93
+      # "work queue depth exceeded" in one day, 2026-10-18). At 1, every slow
+      # block paged. 4 consecutive failed minutes is past any block burst seen,
+      # and a real outage still alerts within ~5 minutes.
+      - {name: bitcoin-knots,    host: knots_box_local, failure_threshold: 4}
       - {name: datum-gateway,    host: knots_box_local}
       - {name: fulcrum,          host: fulcrum_box_local}
       - {name: phoenixd,         host: vipy}
@@ -54,6 +60,16 @@
       - {name: navidrome,        host: music_box_local}
       - {name: jar-lid-archive-staging, host: jar_lid_staging_local}
 
+    # The alert every probe gets. failure-threshold is 1 - the heartbeat window
+    # is the tolerance, see the note above - unless a probe sets
+    # failure_threshold, for checks that legitimately fail for a minute or two.
+    probe_alert:
+      type: signal
+      failure-threshold: 1
+      success-threshold: 2
+      send-on-resolved: true
+      minimum-reminder-interval: 6h
+
   tasks:
     - name: Build the probe endpoint list
       ansible.builtin.set_fact:
@@ -61,19 +77,13 @@
           'name': item.name,
           'group': 'probe',
           'token': gatus_push_tokens[item.host],
-          'heartbeat': item.heartbeat | default('11m')}] }}"
+          'heartbeat': item.heartbeat | default('11m'),
+          'alerts': [probe_alert | combine({'failure-threshold': item.failure_threshold | default(1)})]}] }}"
       loop: "{{ probes }}"
 
     - name: Register the probe endpoints
       ansible.builtin.include_role:
         name: gatus_endpoint
       vars:
-        gatus_endpoint_default_alerts:
-          - type: signal
-            # 1, not 3: the heartbeat window is the tolerance. See the note above.
-            failure-threshold: 1
-            success-threshold: 2
-            send-on-resolved: true
-            minimum-reminder-interval: 6h
         gatus_endpoint_name: probes
         gatus_endpoint_external: "{{ probe_endpoints }}"