monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity

Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:

  infra/410_disk_usage_alerts.yml   -> disk-usage check   (infra/400)
  infra/420_system_healthcheck.yml  -> liveness check     (infra/400)
  infra/430_cpu_temp_alerts.yml     -> cpu-temp check     (infra/400)
  32_zfs play 2 (monitoring half)   -> zfs-health check   (infra/400)
  34_nut play 2 (entirely)          -> ups-status check   (infra/400)

Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.

Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.

A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.

ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.

Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).

site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.

Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
counterweight 2026-09-14 10:18:57 +02:00
parent 99760dfd46
commit 853e62a19c
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
16 changed files with 184 additions and 1520 deletions

View file

@ -18,6 +18,17 @@
- import_playbook: infra/02_firewall_and_fail2ban_playbook.yml
- import_playbook: infra/900_install_rsync.yml
- import_playbook: infra/920_join_headscale_mesh.yml
# Idempotent and kept permanently: guarantees a rebuilt or restored host cannot
# quietly bring the Uptime-Kuma-era monitoring back.
- import_playbook: infra/409_remove_legacy_monitoring.yml
# ── Monitoring ──────────────────────────────────────────────────────────────
# Gatus first: the three plays below register endpoints with it, and registering
# against a host that is not serving yet would simply fail.
- import_playbook: services/gatus/deploy_gatus_playbook.yml
- import_playbook: infra/400_host_monitoring.yml
- import_playbook: infra/401_service_monitoring.yml
- import_playbook: infra/402_public_monitoring.yml
# 910_docker says `hosts: managed`, but only 5 of 11 managed hosts have or need
# Docker. Left out until it has a [docker] group — see the note in PLAN_7.
@ -56,13 +67,6 @@
# Deliberately not here. Every playbook in the repo is either imported above or
# listed below, so this file accounts for all of them:
#
# infra/410_disk_usage_alerts.yml assert on the Uptime Kuma credentials that
# infra/420_system_healthcheck.yml were removed from the vault, so they fail
# infra/430_cpu_temp_alerts.yml before doing anything. What they deploy IS
# running on the boxes - same state the two
# nodito playbooks were in before 0f03c50.
# Add them once they are de-Kuma'd.
#
# infra/910_docker_playbook.yml says `hosts: managed`, but Docker is on 5
# of 11 managed hosts and those 5 are exactly
# the ones that need it. Running it would
@ -74,7 +78,7 @@
#
# services/ntfy/setup_ntfy_uptime_ creates a notification channel INSIDE
# kuma_notification.yml Uptime Kuma. Kuma-specific tooling, not
# a deployment.
# a deployment, and Kuma is being retired.
#
# services/vaultwarden/disable_ deliberate manual actions, not convergence
# vaultwarden_sign_ups_playbook.yml