personal_infra/ansible/roles/healthcheck/templates/checks/zfs-health.sh.j2

64 lines
2.9 KiB
Text
Raw Normal View History

monitoring: recover host checks for the whole estate, reported to Gatus Five checks, 27 endpoints, replacing what Uptime Kuma used to watch: is it up every 5min, all hosts is disk full daily, all hosts is CPU hot every 5min, nodito is ZFS broken daily, nodito is UPS online every 5min, nodito Two roles, kept separate so neither knows about the other - they meet at a URL and a token, the same way caddy_site and each service meet at a vhost: roles/gatus_endpoint runs on the observability host, writes ONE file into /opt/gatus/config/endpoints/. Gatus merges every *.yaml there and appends lists, so callers compose without coordinating. roles/healthcheck runs on the monitored host: a check script, a systemd service, a timer, and an optional push. Ships a library of check bodies under templates/checks/. Everything PUSHES. Gatus never reaches out, which matters because nodito and its VMs are behind NAT, and because four of the five checks are internal state with no pollable surface at all. Liveness pushes too, deliberately: a heartbeat proves the host is running AND can reach the internet, where an ICMP probe from one vantage point only proves it answers pings from there. And since Gatus alerts when a heartbeat window expires, a check that stops running raises the alarm by itself - a dead timer looks exactly like a dead host, which is the correct reading. One bearer token per host, generated straight into the vault and never printed. A token only writes results for its own host's endpoints, so a compromised host can lie about itself, which it could do anyway. Three things learned from the source that shaped this: * Gatus polls its own config every 30s and reloads (main.listenToConfigurationFileChanges), so gatus_endpoint needs no restart handler - writing the file IS the deploy. * ...but on a reload it panics if the new config fails to parse, unless skip-invalid-config-update is set. Endpoint files are contributed by other playbooks, so one malformed file would take the monitor down at the worst possible moment. Now set. * The push URL uses a key Gatus computes, not the name you write: sanitize(group) + "_" + sanitize(name), lowercased with / _ . , space # + & replaced by "-" (config/key/key.go). So knots_box_local is knots-box-local in the URL. The playbook derives it rather than hand-writing. storage: maximum-number-of-results 900, up from upstream's 100. Gatus bounds the database by COUNT and trims inline on insert, so there is no retention job and no way to fill a disk - but history depth is then a function of check frequency, and 100 results at a 5-minute interval is 8 hours. 900 is ~3 days of liveness and ~2.5 years of the daily disk check. The uptime table is separate and its 30-day retention is hard-coded upstream. A bug worth recording: the first deploy shipped five scripts that all died with "syntax error: unexpected end of file". Jinja strips an included template's trailing newline and trim_blocks then eats the newline after {% endif %}, so the closing brace of check() landed on the same line as the body's last statement - `return 0}`. Every check was broken and the deploy still reported failed=0, because the role's "run once" task has failed_when: false and reports the result as a debug message nobody read. The blank line that fixes it is now load-bearing and commented as such. Verified by triggering every unit by hand rather than waiting on timers: all checks exit 0 on all hosts, and Gatus shows 26 UP / 1 DOWN. The one DOWN is liveness_watchtower, which is honest - that host currently refuses SSH (TCP connects, no banner exchange) and is excluded from this deploy. It is also the box still running Uptime Kuma. Known waste, not yet fixed: healthcheck installs its dependencies per CHECK rather than per HOST, so apt runs 29 times estate-wide for a curl that is already present, and daemon_reload runs 4x per host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 08:54:10 +02:00
# Five conditions, all of which have to hold. Ported from the check that
# infra/nodito/32_zfs_pool_setup_playbook.yml deployed, which was correct -
# only its reporting was tied to Uptime Kuma.
local pool="{{ healthcheck_zfs_pool }}"
local json issues=""
json=$(zpool status -j "$pool" 2>&1) || { MESSAGE="zpool status failed: ${json}"; return 1; }
# 1. pool state
local state
state=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].state')
[ "$state" = "ONLINE" ] || issues="${issues}${issues:+; }pool ${state}"
# 2. every vdev and device ONLINE
local bad
bad=$(echo "$json" | jq -r --arg p "$pool" '
.pools[$p].vdevs[] | .. | objects
| select(.state? and .state != "ONLINE")
| "\(.name // "unknown"):\(.state)"' 2>/dev/null | paste -sd, -)
[ -z "$bad" ] || issues="${issues}${issues:+; }devices ${bad}"
# 3. resilver in progress
local fn st
fn=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.function // "NONE"')
st=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.state // "NONE"')
if [ "$fn" = "RESILVER" ] && [ "$st" = "SCANNING" ]; then
issues="${issues}${issues:+; }resilvering"
fi
# 4. read/write/checksum errors. ZFS reports these as strings.
local errs
errs=$(echo "$json" | jq -r --arg p "$pool" '
.pools[$p].vdevs[] | .. | objects
| select(.name? and ((.read_errors // "0" | tonumber) > 0
or (.write_errors // "0" | tonumber) > 0
or (.checksum_errors // "0" | tonumber) > 0))
| "\(.name) r=\(.read_errors) w=\(.write_errors) c=\(.checksum_errors)"' 2>/dev/null | paste -sd, -)
[ -z "$errs" ] || issues="${issues}${issues:+; }errors ${errs}"
# 5. errors from the last scrub
local scan_err
scan_err=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.errors // "0"')
if [ -n "$scan_err" ] && [ "$scan_err" != "0" ] && [ "$scan_err" != "null" ]; then
issues="${issues}${issues:+; }scan errors ${scan_err}"
fi
monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity Five things deprecated, each verified against the DEPLOYED script before being deleted rather than assumed superseded: infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400) infra/420_system_healthcheck.yml -> liveness check (infra/400) infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400) 32_zfs play 2 (monitoring half) -> zfs-health check (infra/400) 34_nut play 2 (entirely) -> ups-status check (infra/400) Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime and pushed, which is exactly a liveness heartbeat. The old disk monitor was WEAKER than its replacement: it checked "/" alone at 80%, where the new one walks every real filesystem at 85%. Deleting the playbooks was not the hard part. The units they installed live on the hosts, enabled, and keep firing regardless of what the repo says - two of them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine machines. A playbook deleted without a cleanup leaves its output running forever with nothing left to explain it. So infra/409_remove_legacy_monitoring stops, disables and removes the units, deletes /opt/{disk-monitoring, system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma reference and reporting it. Kept permanently and idempotent, so a rebuilt or restored host cannot quietly bring them back. A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the play wholesale would have silently stopped scrubbing the pool - and an unscrubbed pool makes the health check meaningless, because it would have nothing true to report. That play is now scrub-only and check-runs ok=5 changed=0. ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read separately with `zpool list` - and it is the failure you get warning of rather than the one you discover. Threshold 80%, because ZFS allocation degrades badly past roughly that and fragmentation is hard to undo. The pool is at 47%. Verified both directions: passes on the real pool, and a simulated 91% exits 1. Also fixed the waste recorded in c2de6db: the healthcheck role installed its dependencies once per CHECK rather than per HOST - 29 apt transactions estate-wide for a curl already present, and the slowest part of every deploy. It now deduplicates within a play run, and the redundant standalone daemon_reload is gone (the systemd task already does one). site.yml updated, which exposed that services/gatus was never in it. It now runs before the three registration playbooks, since registering endpoints against a Gatus that is not yet serving would simply fail. Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 10:18:57 +02:00
# 6. Capacity. Not an error condition in `zpool status` - a 95% full pool is
# reported perfectly ONLINE - so it has to be read separately, and it is
# the failure you get warning of rather than the one you discover.
local capacity
capacity=$(zpool list -H -o capacity "$pool" 2>/dev/null | tr -dc '0-9')
if [ -z "$capacity" ]; then
issues="${issues}${issues:+; }cannot read capacity"
elif [ "$capacity" -ge {{ healthcheck_zfs_capacity_threshold }} ]; then
issues="${issues}${issues:+; }pool ${capacity}% full (>={{ healthcheck_zfs_capacity_threshold }}%)"
fi
monitoring: recover host checks for the whole estate, reported to Gatus Five checks, 27 endpoints, replacing what Uptime Kuma used to watch: is it up every 5min, all hosts is disk full daily, all hosts is CPU hot every 5min, nodito is ZFS broken daily, nodito is UPS online every 5min, nodito Two roles, kept separate so neither knows about the other - they meet at a URL and a token, the same way caddy_site and each service meet at a vhost: roles/gatus_endpoint runs on the observability host, writes ONE file into /opt/gatus/config/endpoints/. Gatus merges every *.yaml there and appends lists, so callers compose without coordinating. roles/healthcheck runs on the monitored host: a check script, a systemd service, a timer, and an optional push. Ships a library of check bodies under templates/checks/. Everything PUSHES. Gatus never reaches out, which matters because nodito and its VMs are behind NAT, and because four of the five checks are internal state with no pollable surface at all. Liveness pushes too, deliberately: a heartbeat proves the host is running AND can reach the internet, where an ICMP probe from one vantage point only proves it answers pings from there. And since Gatus alerts when a heartbeat window expires, a check that stops running raises the alarm by itself - a dead timer looks exactly like a dead host, which is the correct reading. One bearer token per host, generated straight into the vault and never printed. A token only writes results for its own host's endpoints, so a compromised host can lie about itself, which it could do anyway. Three things learned from the source that shaped this: * Gatus polls its own config every 30s and reloads (main.listenToConfigurationFileChanges), so gatus_endpoint needs no restart handler - writing the file IS the deploy. * ...but on a reload it panics if the new config fails to parse, unless skip-invalid-config-update is set. Endpoint files are contributed by other playbooks, so one malformed file would take the monitor down at the worst possible moment. Now set. * The push URL uses a key Gatus computes, not the name you write: sanitize(group) + "_" + sanitize(name), lowercased with / _ . , space # + & replaced by "-" (config/key/key.go). So knots_box_local is knots-box-local in the URL. The playbook derives it rather than hand-writing. storage: maximum-number-of-results 900, up from upstream's 100. Gatus bounds the database by COUNT and trims inline on insert, so there is no retention job and no way to fill a disk - but history depth is then a function of check frequency, and 100 results at a 5-minute interval is 8 hours. 900 is ~3 days of liveness and ~2.5 years of the daily disk check. The uptime table is separate and its 30-day retention is hard-coded upstream. A bug worth recording: the first deploy shipped five scripts that all died with "syntax error: unexpected end of file". Jinja strips an included template's trailing newline and trim_blocks then eats the newline after {% endif %}, so the closing brace of check() landed on the same line as the body's last statement - `return 0}`. Every check was broken and the deploy still reported failed=0, because the role's "run once" task has failed_when: false and reports the result as a debug message nobody read. The blank line that fixes it is now load-bearing and commented as such. Verified by triggering every unit by hand rather than waiting on timers: all checks exit 0 on all hosts, and Gatus shows 26 UP / 1 DOWN. The one DOWN is liveness_watchtower, which is honest - that host currently refuses SSH (TCP connects, no banner exchange) and is excluded from this deploy. It is also the box still running Uptime Kuma. Known waste, not yet fixed: healthcheck installs its dependencies per CHECK rather than per HOST, so apt runs 29 times estate-wide for a curl that is already present, and daemon_reload runs 4x per host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 08:54:10 +02:00
if [ -n "$issues" ]; then MESSAGE="$issues"; return 1; fi
local scrub
scrub=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.start_time // "never"')
monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity Five things deprecated, each verified against the DEPLOYED script before being deleted rather than assumed superseded: infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400) infra/420_system_healthcheck.yml -> liveness check (infra/400) infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400) 32_zfs play 2 (monitoring half) -> zfs-health check (infra/400) 34_nut play 2 (entirely) -> ups-status check (infra/400) Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime and pushed, which is exactly a liveness heartbeat. The old disk monitor was WEAKER than its replacement: it checked "/" alone at 80%, where the new one walks every real filesystem at 85%. Deleting the playbooks was not the hard part. The units they installed live on the hosts, enabled, and keep firing regardless of what the repo says - two of them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine machines. A playbook deleted without a cleanup leaves its output running forever with nothing left to explain it. So infra/409_remove_legacy_monitoring stops, disables and removes the units, deletes /opt/{disk-monitoring, system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma reference and reporting it. Kept permanently and idempotent, so a rebuilt or restored host cannot quietly bring them back. A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the play wholesale would have silently stopped scrubbing the pool - and an unscrubbed pool makes the health check meaningless, because it would have nothing true to report. That play is now scrub-only and check-runs ok=5 changed=0. ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read separately with `zpool list` - and it is the failure you get warning of rather than the one you discover. Threshold 80%, because ZFS allocation degrades badly past roughly that and fragmentation is hard to undo. The pool is at 47%. Verified both directions: passes on the real pool, and a simulated 91% exits 1. Also fixed the waste recorded in c2de6db: the healthcheck role installed its dependencies once per CHECK rather than per HOST - 29 apt transactions estate-wide for a curl already present, and the slowest part of every deploy. It now deduplicates within a play run, and the redundant standalone daemon_reload is gone (the systemd task already does one). site.yml updated, which exposed that services/gatus was never in it. It now runs before the three registration playbooks, since registering endpoints against a Gatus that is not yet serving would simply fail. Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 10:18:57 +02:00
MESSAGE="${pool} ONLINE, ${capacity}% full, last scrub ${scrub}"
monitoring: recover host checks for the whole estate, reported to Gatus Five checks, 27 endpoints, replacing what Uptime Kuma used to watch: is it up every 5min, all hosts is disk full daily, all hosts is CPU hot every 5min, nodito is ZFS broken daily, nodito is UPS online every 5min, nodito Two roles, kept separate so neither knows about the other - they meet at a URL and a token, the same way caddy_site and each service meet at a vhost: roles/gatus_endpoint runs on the observability host, writes ONE file into /opt/gatus/config/endpoints/. Gatus merges every *.yaml there and appends lists, so callers compose without coordinating. roles/healthcheck runs on the monitored host: a check script, a systemd service, a timer, and an optional push. Ships a library of check bodies under templates/checks/. Everything PUSHES. Gatus never reaches out, which matters because nodito and its VMs are behind NAT, and because four of the five checks are internal state with no pollable surface at all. Liveness pushes too, deliberately: a heartbeat proves the host is running AND can reach the internet, where an ICMP probe from one vantage point only proves it answers pings from there. And since Gatus alerts when a heartbeat window expires, a check that stops running raises the alarm by itself - a dead timer looks exactly like a dead host, which is the correct reading. One bearer token per host, generated straight into the vault and never printed. A token only writes results for its own host's endpoints, so a compromised host can lie about itself, which it could do anyway. Three things learned from the source that shaped this: * Gatus polls its own config every 30s and reloads (main.listenToConfigurationFileChanges), so gatus_endpoint needs no restart handler - writing the file IS the deploy. * ...but on a reload it panics if the new config fails to parse, unless skip-invalid-config-update is set. Endpoint files are contributed by other playbooks, so one malformed file would take the monitor down at the worst possible moment. Now set. * The push URL uses a key Gatus computes, not the name you write: sanitize(group) + "_" + sanitize(name), lowercased with / _ . , space # + & replaced by "-" (config/key/key.go). So knots_box_local is knots-box-local in the URL. The playbook derives it rather than hand-writing. storage: maximum-number-of-results 900, up from upstream's 100. Gatus bounds the database by COUNT and trims inline on insert, so there is no retention job and no way to fill a disk - but history depth is then a function of check frequency, and 100 results at a 5-minute interval is 8 hours. 900 is ~3 days of liveness and ~2.5 years of the daily disk check. The uptime table is separate and its 30-day retention is hard-coded upstream. A bug worth recording: the first deploy shipped five scripts that all died with "syntax error: unexpected end of file". Jinja strips an included template's trailing newline and trim_blocks then eats the newline after {% endif %}, so the closing brace of check() landed on the same line as the body's last statement - `return 0}`. Every check was broken and the deploy still reported failed=0, because the role's "run once" task has failed_when: false and reports the result as a debug message nobody read. The blank line that fixes it is now load-bearing and commented as such. Verified by triggering every unit by hand rather than waiting on timers: all checks exit 0 on all hosts, and Gatus shows 26 UP / 1 DOWN. The one DOWN is liveness_watchtower, which is honest - that host currently refuses SSH (TCP connects, no banner exchange) and is excluded from this deploy. It is also the box still running Uptime Kuma. Known waste, not yet fixed: healthcheck installs its dependencies per CHECK rather than per HOST, so apt runs 29 times estate-wide for a curl that is already present, and daemon_reload runs 4x per host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 08:54:10 +02:00
return 0