counterweight
853e62a19c
Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
54 lines
2.4 KiB
YAML
54 lines
2.4 KiB
YAML
---
|
|
# One health check: a script, a systemd service, a timer, and an optional push.
|
|
#
|
|
# The exit code is the answer and systemd keeps it:
|
|
# systemctl is-failed <name>-healthcheck.service
|
|
# Reporting anywhere else is optional and generic. Point healthcheck_push_url at
|
|
# Gatus, or at whatever replaces it, or at nothing.
|
|
|
|
healthcheck_name: "" # e.g. disk-usage -> disk-usage-healthcheck
|
|
healthcheck_description: ""
|
|
|
|
# The check itself. Pick ONE:
|
|
# healthcheck_check: a template under templates/checks/ (without .sh.j2)
|
|
# healthcheck_command: a shell one-liner that exits 0 for healthy
|
|
healthcheck_check: ""
|
|
healthcheck_command: ""
|
|
|
|
# systemd timer. OnUnitActiveSec unless healthcheck_on_calendar is set.
|
|
healthcheck_interval: "5min"
|
|
healthcheck_on_calendar: ""
|
|
healthcheck_boot_delay: "2min"
|
|
|
|
# ── Reporting ────────────────────────────────────────────────────────────────
|
|
# Gatus external endpoints:
|
|
# POST {url}?success=true|false&error=...
|
|
# Authorization: Bearer {token}
|
|
# Empty url = check and log only, which is a valid state and not an error.
|
|
healthcheck_push_url: ""
|
|
healthcheck_push_token: ""
|
|
|
|
# Some checks report MORE THAN ONE result - a host with four systemd services
|
|
# needs four endpoints, or a single red light cannot tell you which one died.
|
|
# Those set healthcheck_push_base to the endpoints COLLECTION and the check body
|
|
# appends each key itself, the same way check-backups.sh reports per source.
|
|
healthcheck_push_base: ""
|
|
|
|
# Units for the systemd-units check. Each becomes its own Gatus endpoint.
|
|
healthcheck_units: []
|
|
# Prefix for the per-unit endpoint keys, e.g. "services_vipy" -> services_vipy-caddy.
|
|
healthcheck_units_key_prefix: ""
|
|
|
|
healthcheck_script_dir: /usr/local/bin
|
|
healthcheck_log_dir: /var/log/healthchecks
|
|
|
|
# Per-check knobs, consumed by the templates under checks/
|
|
healthcheck_disk_threshold: 85 # percent
|
|
healthcheck_cpu_temp_threshold: 80 # celsius
|
|
healthcheck_zfs_pool: ""
|
|
# ZFS degrades badly once a pool passes roughly 80% - allocation gets slow and
|
|
# fragmentation becomes hard to undo, and unlike a normal filesystem you cannot
|
|
# simply delete your way back to good performance. So this alarms well before
|
|
# the pool is actually out of space.
|
|
healthcheck_zfs_capacity_threshold: 80
|
|
healthcheck_ups_name: ""
|