From 0112bb2c69936f1773af11d71e1732760bf253c2 Mon Sep 17 00:00:00 2001 From: counterweight Date: Tue, 15 Sep 2026 20:42:59 +0200 Subject: [PATCH] disk check: it never worked. Fix it, and the false DOWNs it hid behind MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Investigating a dashboard full of DOWN services that were plainly healthy turned up three bugs, all introduced by me. ── 1. The disk check has been lying since it was written ─────────────────── df: options -P and --output are mutually exclusive `df -P ... --output=pcent,target` errors out. `2>/dev/null` swallowed it, the read loop got nothing, `worst` stayed 0, and the check reported "max 0% on /" and exited 0 on EVERY host regardless of real usage. vipy is at 58%. It would never have caught a full disk - a green light wired to nothing, which is worse than no check at all. Dropped -P. More importantly, "df returned no filesystems" is now a FAILURE rather than being read as 0% - the original bug was only invisible because an empty result was indistinguishable from an empty disk. Testing the fix immediately surfaced a second problem it had been masking: /sys/firmware/efi/efivars sits at 87% on a healthy machine, so a working check would have paged daily. efivarfs and ramfs join tmpfs/devtmpfs/squashfs/overlay in the exclusions, all of them pseudo-filesystems whose "usage" is not a fact an operator can act on. It now reports "max 73% on / (3 filesystems)". ── 2. The false DOWNs were a half-finished deploy ────────────────────────── The previous commit tightened heartbeats from 30h to 7h AND moved the checks from daily to 6-hourly - but only the registration side was deployed, with --limit observability. The hosts kept pushing once a day against a 7h window, so after seven hours every disk, ZFS and backup-store endpoint went red. Nothing was wrong with the services and nothing was wrong with Gatus: it correctly reported that pushes were not arriving. Changing a heartbeat window without deploying the matching cadence is a guaranteed false alarm, and the two have to ship together. ── 3. The new OnCalendar was invalid ─────────────────────────────────────── "*-*-* 05:30:00,11:30:00,17:30:00,23:30:00" is not valid systemd syntax - a comma-separated list of FULL TIMES is rejected with "bad unit file setting", and the timer silently failed to install. Corrected to "*-*-* 05/6:30:00" and validated with `systemd-analyze calendar` before deploying, which is how this should have been written in the first place. Verified: 11 disk checks and the ZFS check all HEALTHY on a manual run; timers confirmed on the hosts as 00/6:00:00, 00/6:20:00 and 05/6:30:00; 86 UP / 0 DOWN. Co-Authored-By: Claude Opus 5 (1M context) --- ansible/roles/backup_store/defaults/main.yml | 5 +++- .../templates/checks/disk-usage.sh.j2 | 28 +++++++++++++++---- 2 files changed, 26 insertions(+), 7 deletions(-) diff --git a/ansible/roles/backup_store/defaults/main.yml b/ansible/roles/backup_store/defaults/main.yml index e6d17a3..9032d55 100644 --- a/ansible/roles/backup_store/defaults/main.yml +++ b/ansible/roles/backup_store/defaults/main.yml @@ -22,7 +22,10 @@ backup_store_check_push_token: "" # source's dump timestamp out of the artefact filename, so running it more often # catches "the source stopped dumping" within hours rather than a day, and lets # the Gatus heartbeat be 7h instead of 30h. -backup_store_check_on_calendar: "*-*-* 05:30:00,11:30:00,17:30:00,23:30:00" +# 05:30 then every 6h. Note the syntax: a comma-separated list of full +# times ("05:30:00,11:30:00,...") is NOT valid and systemd rejects the unit +# with "bad unit file setting" - validated with `systemd-analyze calendar`. +backup_store_check_on_calendar: "*-*-* 05/6:30:00" # An artefact older than this is stale. Sources dump daily at 02:00-02:30 and the # pull is at 04:00, so 26h tolerates exactly one missed night before alarming. diff --git a/ansible/roles/healthcheck/templates/checks/disk-usage.sh.j2 b/ansible/roles/healthcheck/templates/checks/disk-usage.sh.j2 index b0e76ce..86fd3a6 100644 --- a/ansible/roles/healthcheck/templates/checks/disk-usage.sh.j2 +++ b/ansible/roles/healthcheck/templates/checks/disk-usage.sh.j2 @@ -1,19 +1,35 @@ - # Every real filesystem must be under the threshold. tmpfs, devtmpfs, - # squashfs and overlay are excluded: they are either RAM, read-only, or - # container layers, and none of them fills up in a way an operator can act on. + # Every real filesystem must be under the threshold. The exclusions are all + # pseudo-filesystems whose "usage" is not a disk-space fact an operator can + # act on: tmpfs and devtmpfs are RAM, squashfs is read-only, overlay is a + # container layer, and efivarfs is UEFI variable storage - which sits near + # 90% full on a perfectly healthy machine and would page every single day. + # + # NOTE: no -P. `df -P --output=...` fails with "options -P and --output are + # mutually exclusive", and an earlier version of this check had both - so df + # errored, the loop read nothing, and it reported "max 0% on /" and exited 0 + # on every host regardless of real usage. A disk check that always says 0% + # is worse than no disk check, so the guard below treats "no filesystems + # read" as a failure rather than as health. local threshold={{ healthcheck_disk_threshold }} - local worst=0 worst_mount="" over="" + local worst=0 worst_mount="" over="" seen=0 while read -r pct mount; do pct=${pct%\%} [ -z "$pct" ] && continue + case "$pct" in (*[!0-9]*) continue;; esac + seen=$((seen + 1)) if [ "$pct" -gt "$worst" ]; then worst=$pct; worst_mount=$mount; fi if [ "$pct" -ge "$threshold" ]; then over="${over}${over:+, }${mount} ${pct}%"; fi - done < <(df -P -x tmpfs -x devtmpfs -x squashfs -x overlay --output=pcent,target 2>/dev/null | tail -n +2) + done < <(df -x tmpfs -x devtmpfs -x squashfs -x overlay -x efivarfs -x ramfs --output=pcent,target 2>/dev/null | tail -n +2) + + if [ "$seen" -eq 0 ]; then + MESSAGE="df returned no filesystems - the check itself is broken" + return 1 + fi if [ -n "$over" ]; then MESSAGE="over ${threshold}%: ${over}" return 1 fi - MESSAGE="max ${worst}% on ${worst_mount:-/}" + MESSAGE="max ${worst}% on ${worst_mount:-/} (${seen} filesystems)" return 0