counterweight
853e62a19c
Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
63 lines
2.9 KiB
Django/Jinja
63 lines
2.9 KiB
Django/Jinja
# Five conditions, all of which have to hold. Ported from the check that
|
|
# infra/nodito/32_zfs_pool_setup_playbook.yml deployed, which was correct -
|
|
# only its reporting was tied to Uptime Kuma.
|
|
local pool="{{ healthcheck_zfs_pool }}"
|
|
local json issues=""
|
|
|
|
json=$(zpool status -j "$pool" 2>&1) || { MESSAGE="zpool status failed: ${json}"; return 1; }
|
|
|
|
# 1. pool state
|
|
local state
|
|
state=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].state')
|
|
[ "$state" = "ONLINE" ] || issues="${issues}${issues:+; }pool ${state}"
|
|
|
|
# 2. every vdev and device ONLINE
|
|
local bad
|
|
bad=$(echo "$json" | jq -r --arg p "$pool" '
|
|
.pools[$p].vdevs[] | .. | objects
|
|
| select(.state? and .state != "ONLINE")
|
|
| "\(.name // "unknown"):\(.state)"' 2>/dev/null | paste -sd, -)
|
|
[ -z "$bad" ] || issues="${issues}${issues:+; }devices ${bad}"
|
|
|
|
# 3. resilver in progress
|
|
local fn st
|
|
fn=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.function // "NONE"')
|
|
st=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.state // "NONE"')
|
|
if [ "$fn" = "RESILVER" ] && [ "$st" = "SCANNING" ]; then
|
|
issues="${issues}${issues:+; }resilvering"
|
|
fi
|
|
|
|
# 4. read/write/checksum errors. ZFS reports these as strings.
|
|
local errs
|
|
errs=$(echo "$json" | jq -r --arg p "$pool" '
|
|
.pools[$p].vdevs[] | .. | objects
|
|
| select(.name? and ((.read_errors // "0" | tonumber) > 0
|
|
or (.write_errors // "0" | tonumber) > 0
|
|
or (.checksum_errors // "0" | tonumber) > 0))
|
|
| "\(.name) r=\(.read_errors) w=\(.write_errors) c=\(.checksum_errors)"' 2>/dev/null | paste -sd, -)
|
|
[ -z "$errs" ] || issues="${issues}${issues:+; }errors ${errs}"
|
|
|
|
# 5. errors from the last scrub
|
|
local scan_err
|
|
scan_err=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.errors // "0"')
|
|
if [ -n "$scan_err" ] && [ "$scan_err" != "0" ] && [ "$scan_err" != "null" ]; then
|
|
issues="${issues}${issues:+; }scan errors ${scan_err}"
|
|
fi
|
|
|
|
# 6. Capacity. Not an error condition in `zpool status` - a 95% full pool is
|
|
# reported perfectly ONLINE - so it has to be read separately, and it is
|
|
# the failure you get warning of rather than the one you discover.
|
|
local capacity
|
|
capacity=$(zpool list -H -o capacity "$pool" 2>/dev/null | tr -dc '0-9')
|
|
if [ -z "$capacity" ]; then
|
|
issues="${issues}${issues:+; }cannot read capacity"
|
|
elif [ "$capacity" -ge {{ healthcheck_zfs_capacity_threshold }} ]; then
|
|
issues="${issues}${issues:+; }pool ${capacity}% full (>={{ healthcheck_zfs_capacity_threshold }}%)"
|
|
fi
|
|
|
|
if [ -n "$issues" ]; then MESSAGE="$issues"; return 1; fi
|
|
|
|
local scrub
|
|
scrub=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.start_time // "never"')
|
|
MESSAGE="${pool} ONLINE, ${capacity}% full, last scrub ${scrub}"
|
|
return 0
|