counterweight
0112bb2c69
Investigating a dashboard full of DOWN services that were plainly healthy turned
up three bugs, all introduced by me.
── 1. The disk check has been lying since it was written ───────────────────
df: options -P and --output are mutually exclusive
`df -P ... --output=pcent,target` errors out. `2>/dev/null` swallowed it, the
read loop got nothing, `worst` stayed 0, and the check reported "max 0% on /"
and exited 0 on EVERY host regardless of real usage. vipy is at 58%. It would
never have caught a full disk - a green light wired to nothing, which is worse
than no check at all.
Dropped -P. More importantly, "df returned no filesystems" is now a FAILURE
rather than being read as 0% - the original bug was only invisible because an
empty result was indistinguishable from an empty disk.
Testing the fix immediately surfaced a second problem it had been masking:
/sys/firmware/efi/efivars sits at 87% on a healthy machine, so a working check
would have paged daily. efivarfs and ramfs join tmpfs/devtmpfs/squashfs/overlay
in the exclusions, all of them pseudo-filesystems whose "usage" is not a fact an
operator can act on. It now reports "max 73% on / (3 filesystems)".
── 2. The false DOWNs were a half-finished deploy ──────────────────────────
The previous commit tightened heartbeats from 30h to 7h AND moved the checks
from daily to 6-hourly - but only the registration side was deployed, with
--limit observability. The hosts kept pushing once a day against a 7h window, so
after seven hours every disk, ZFS and backup-store endpoint went red.
Nothing was wrong with the services and nothing was wrong with Gatus: it
correctly reported that pushes were not arriving. Changing a heartbeat window
without deploying the matching cadence is a guaranteed false alarm, and the two
have to ship together.
── 3. The new OnCalendar was invalid ───────────────────────────────────────
"*-*-* 05:30:00,11:30:00,17:30:00,23:30:00" is not valid systemd syntax - a
comma-separated list of FULL TIMES is rejected with "bad unit file setting", and
the timer silently failed to install. Corrected to "*-*-* 05/6:30:00" and
validated with `systemd-analyze calendar` before deploying, which is how this
should have been written in the first place.
Verified: 11 disk checks and the ZFS check all HEALTHY on a manual run; timers
confirmed on the hosts as 00/6:00:00, 00/6:20:00 and 05/6:30:00; 86 UP / 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
32 lines
1.7 KiB
YAML
32 lines
1.7 KiB
YAML
---
|
|
backup_store_dir: "{{ ansible_env.HOME }}/backups"
|
|
backup_store_ssh_key: "{{ ansible_env.HOME }}/.ssh/id_pull"
|
|
backup_store_on_calendar: "*-*-* 04:00:00"
|
|
|
|
# One entry per source. `retention_days` is the LONG tail; the source keeps its
|
|
# own short local retention.
|
|
# - name: headscale
|
|
# source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/"
|
|
# retention_days: 90
|
|
backup_store_sources: []
|
|
|
|
# ── Reporting ────────────────────────────────────────────────────────────────
|
|
# check-backups.sh reports one result PER SOURCE plus one for the store itself,
|
|
# so the base URL is the endpoints collection and the script appends each key.
|
|
# Empty is valid: the script still prints its report and exits 0/1.
|
|
backup_store_check_push_base: ""
|
|
backup_store_check_push_token: ""
|
|
|
|
# Every six hours, offset past the 04:00 pull so the first run of the day sees a
|
|
# finished pull. The BACKUPS are daily, but this check is not - it reads the
|
|
# source's dump timestamp out of the artefact filename, so running it more often
|
|
# catches "the source stopped dumping" within hours rather than a day, and lets
|
|
# the Gatus heartbeat be 7h instead of 30h.
|
|
# 05:30 then every 6h. Note the syntax: a comma-separated list of full
|
|
# times ("05:30:00,11:30:00,...") is NOT valid and systemd rejects the unit
|
|
# with "bad unit file setting" - validated with `systemd-analyze calendar`.
|
|
backup_store_check_on_calendar: "*-*-* 05/6:30:00"
|
|
|
|
# An artefact older than this is stale. Sources dump daily at 02:00-02:30 and the
|
|
# pull is at 04:00, so 26h tolerates exactly one missed night before alarming.
|
|
backup_store_check_max_age_hours: 26
|