counterweight
f33f1a4b67
The probes, the unit check and the public checks ran once a day with 26h windows, justified as "a multi-week initial sync on a rotating disk". That stopped being true: the node is synced, and a cold explorer page now renders in 29s and a warm one in 0.2s - polling every 5 minutes is what keeps it warm. A dead monerod could go unnoticed for a day. Back to the standard 5m cadence and 11m heartbeat; the per-host service_health override on monerod-box is gone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
79 lines
3.8 KiB
YAML
79 lines
3.8 KiB
YAML
---
|
|
# The per-service health probes.
|
|
#
|
|
# These are NOT the same thing as the systemd checks in infra/401. Those answer
|
|
# "is the unit running"; these answer "does the service actually respond" - an
|
|
# RPC call to bitcoind, a TCP connect to Fulcrum's Electrum port, an HTTP fetch
|
|
# from the Mempool backend. A process can be perfectly `active` and useless,
|
|
# which is precisely the gap these close.
|
|
#
|
|
# The checks themselves live in each service's own role, deployed by that
|
|
# service's playbook. This play only registers where they report, because the
|
|
# endpoints must exist in Gatus before the first push arrives.
|
|
#
|
|
# They used to push to Uptime Kuma. The scripts now POST with a bearer token
|
|
# instead of GETting ?status=up, and each host uses its own token.
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Alerting thresholds, and why they differ by check type.
|
|
#
|
|
# `failure-threshold` counts CONSECUTIVE failures, but "consecutive" means a
|
|
# different amount of wall-clock time per check:
|
|
#
|
|
# push/heartbeat endpoints a failure is produced once per heartbeat window
|
|
# pulled endpoints a failure is produced once per interval
|
|
#
|
|
# So the default of 3 would mean 33 minutes on an 11m heartbeat and over a day
|
|
# on a 7h one - and the heartbeat window ALREADY encodes the tolerance. An 11m
|
|
# window on a 5-minute push is precisely "one missed push forgiven"; stacking a
|
|
# threshold of 3 on top triples a tolerance that was already chosen.
|
|
#
|
|
# Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints
|
|
# have no built-in tolerance, so the threshold is where it belongs for them.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
- name: Register the per-service probes with Gatus
|
|
hosts: observability
|
|
become: yes
|
|
|
|
vars:
|
|
probes:
|
|
- {name: bitcoin-knots, host: knots_box_local}
|
|
- {name: datum-gateway, host: knots_box_local}
|
|
- {name: fulcrum, host: fulcrum_box_local}
|
|
- {name: phoenixd, host: vipy}
|
|
- {name: forgejo-runner, host: forgejo_runner_local}
|
|
- {name: mempool-mariadb, host: mempool_box_local}
|
|
- {name: mempool-backend, host: mempool_box_local}
|
|
- {name: mempool-frontend, host: mempool_box_local}
|
|
# Every 5 minutes like the rest since 2026-10-17 (they were daily while
|
|
# the node synced on a rotating disk). Cadence lives in
|
|
# roles/{monero,xmrblocks}/defaults/main.yml; it and this window are one
|
|
# decision and must move together.
|
|
- {name: monerod, host: monerod_box_local}
|
|
- {name: xmrblocks, host: monerod_box_local}
|
|
- {name: navidrome, host: music_box_local}
|
|
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
|
|
|
|
tasks:
|
|
- name: Build the probe endpoint list
|
|
ansible.builtin.set_fact:
|
|
probe_endpoints: "{{ probe_endpoints | default([]) + [{
|
|
'name': item.name,
|
|
'group': 'probe',
|
|
'token': gatus_push_tokens[item.host],
|
|
'heartbeat': item.heartbeat | default('11m')}] }}"
|
|
loop: "{{ probes }}"
|
|
|
|
- name: Register the probe endpoints
|
|
ansible.builtin.include_role:
|
|
name: gatus_endpoint
|
|
vars:
|
|
gatus_endpoint_default_alerts:
|
|
- type: signal
|
|
# 1, not 3: the heartbeat window is the tolerance. See the note above.
|
|
failure-threshold: 1
|
|
success-threshold: 2
|
|
send-on-resolved: true
|
|
minimum-reminder-interval: 6h
|
|
gatus_endpoint_name: probes
|
|
gatus_endpoint_external: "{{ probe_endpoints }}"
|