counterweight
361683681e
A wedged explorer produced 83 Signal messages in six hours. Nothing about that
was a monitoring success: the checks were right, the cadence was wrong.
monerod-box is a Monero node in the middle of a multi-week initial sync on a
rotating disk. A 5-minute poll tells you nothing a daily one does not, and when
something breaks it turns one fault into a message every five minutes until
somebody silences the phone - which is the failure mode that makes people stop
reading alerts entirely.
So both Monero probes move to OnCalendar:
monerod *-*-* 05:00:00 RandomizedDelaySec=900
xmrblocks *-*-* 05:30:00 RandomizedDelaySec=900
systemd units (infra/401 per-host timer) *-*-* 05:15:00
30 minutes apart on purpose: both read the same LMDB, and the explorer's check
renders a full page out of it - a cold one measured 68 seconds. Running them
together would have each paying the other's cache misses.
OnCalendar rather than OnUnitActiveSec: an interval timer re-bases on every
restart, so "daily" silently becomes "a day after whenever it last came up".
── The window and the cadence are ONE decision ─────────────────────────────
The Gatus heartbeat moves with them, to 26h. Not 24h: the window must EXCEED
the push interval, or a check reporting exactly on schedule races its own
deadline and alerts on jitter alone. With a 900s randomized delay the true
worst case is ~24h15m, so 26h leaves real headroom.
The same logic drops the pulled public-site checks to a 24h interval - and
their failure-threshold from 3 to 1 with it. Three failures at 5m is 15 minutes
of tolerance; three at 24h would be THREE DAYS before anyone heard. A daily
check has no room for a blip, so it alerts on the first.
── Scoping ────────────────────────────────────────────────────────────────
infra/401's timer is ONE PER HOST and iterates every unit in
monitored_services, so slowing it slows all of them. That is safe here only
because monerod-box runs nothing but Monero units; the knobs are therefore in
its host_vars rather than the role, and both 401 and 403 now take a per-host
(resp. per-probe) override with the old 11m as the default. Every other host
is untouched.
Host liveness and disk are deliberately NOT slowed. They answer "is the box
alive", which is still wanted within minutes.
The role-level OnCalendar defaults are prefixed (monero_healthcheck_on_calendar,
xmrblocks_healthcheck_on_calendar) because the generic `healthcheck` role owns
the unprefixed name, and an unprefixed default here would shadow it for any
play that includes both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
80 lines
4 KiB
YAML
80 lines
4 KiB
YAML
---
|
|
# The per-service health probes.
|
|
#
|
|
# These are NOT the same thing as the systemd checks in infra/401. Those answer
|
|
# "is the unit running"; these answer "does the service actually respond" - an
|
|
# RPC call to bitcoind, a TCP connect to Fulcrum's Electrum port, an HTTP fetch
|
|
# from the Mempool backend. A process can be perfectly `active` and useless,
|
|
# which is precisely the gap these close.
|
|
#
|
|
# The checks themselves live in each service's own role, deployed by that
|
|
# service's playbook. This play only registers where they report, because the
|
|
# endpoints must exist in Gatus before the first push arrives.
|
|
#
|
|
# They used to push to Uptime Kuma. The scripts now POST with a bearer token
|
|
# instead of GETting ?status=up, and each host uses its own token.
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# Alerting thresholds, and why they differ by check type.
|
|
#
|
|
# `failure-threshold` counts CONSECUTIVE failures, but "consecutive" means a
|
|
# different amount of wall-clock time per check:
|
|
#
|
|
# push/heartbeat endpoints a failure is produced once per heartbeat window
|
|
# pulled endpoints a failure is produced once per interval
|
|
#
|
|
# So the default of 3 would mean 33 minutes on an 11m heartbeat and over a day
|
|
# on a 7h one - and the heartbeat window ALREADY encodes the tolerance. An 11m
|
|
# window on a 5-minute push is precisely "one missed push forgiven"; stacking a
|
|
# threshold of 3 on top triples a tolerance that was already chosen.
|
|
#
|
|
# Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints
|
|
# have no built-in tolerance, so the threshold is where it belongs for them.
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
- name: Register the per-service probes with Gatus
|
|
hosts: observability
|
|
become: yes
|
|
|
|
vars:
|
|
probes:
|
|
- {name: bitcoin-knots, host: knots_box_local}
|
|
- {name: datum-gateway, host: knots_box_local}
|
|
- {name: fulcrum, host: fulcrum_box_local}
|
|
- {name: phoenixd, host: vipy}
|
|
- {name: forgejo-runner, host: forgejo_runner_local}
|
|
- {name: mempool-mariadb, host: mempool_box_local}
|
|
- {name: mempool-backend, host: mempool_box_local}
|
|
- {name: mempool-frontend, host: mempool_box_local}
|
|
# The two Monero probes push ONCE A DAY, not every 5 minutes, so they
|
|
# need a window wider than a day. 26h, not 24h: the window must exceed
|
|
# the push interval or a check that reports exactly on time races its own
|
|
# deadline and alerts on jitter. The timers are OnCalendar 05:00 and
|
|
# 05:30 with RandomizedDelaySec=900, so the true worst case is ~24h15m.
|
|
# Cadence lives in roles/{monero,xmrblocks}/defaults/main.yml; the two
|
|
# are one decision and must move together.
|
|
- {name: monerod, host: monerod_box_local, heartbeat: '26h'}
|
|
- {name: xmrblocks, host: monerod_box_local, heartbeat: '26h'}
|
|
|
|
tasks:
|
|
- name: Build the probe endpoint list
|
|
ansible.builtin.set_fact:
|
|
probe_endpoints: "{{ probe_endpoints | default([]) + [{
|
|
'name': item.name,
|
|
'group': 'probe',
|
|
'token': gatus_push_tokens[item.host],
|
|
'heartbeat': item.heartbeat | default('11m')}] }}"
|
|
loop: "{{ probes }}"
|
|
|
|
- name: Register the probe endpoints
|
|
ansible.builtin.include_role:
|
|
name: gatus_endpoint
|
|
vars:
|
|
gatus_endpoint_default_alerts:
|
|
- type: signal
|
|
# 1, not 3: the heartbeat window is the tolerance. See the note above.
|
|
failure-threshold: 1
|
|
success-threshold: 2
|
|
send-on-resolved: true
|
|
minimum-reminder-interval: 6h
|
|
gatus_endpoint_name: probes
|
|
gatus_endpoint_external: "{{ probe_endpoints }}"
|