monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity

Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:

  infra/410_disk_usage_alerts.yml   -> disk-usage check   (infra/400)
  infra/420_system_healthcheck.yml  -> liveness check     (infra/400)
  infra/430_cpu_temp_alerts.yml     -> cpu-temp check     (infra/400)
  32_zfs play 2 (monitoring half)   -> zfs-health check   (infra/400)
  34_nut play 2 (entirely)          -> ups-status check   (infra/400)

Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.

Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.

A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.

ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.

Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).

site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.

Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/healthcheck/defaults/main.yml

@@ -46,4 +46,9 @@ healthcheck_log_dir: /var/log/healthchecks
 healthcheck_disk_threshold: 85       # percent
 healthcheck_cpu_temp_threshold: 80   # celsius
 healthcheck_zfs_pool: ""
+# ZFS degrades badly once a pool passes roughly 80% - allocation gets slow and
+# fragmentation becomes hard to undo, and unlike a normal filesystem you cannot
+# simply delete your way back to good performance. So this alarms well before
+# the pool is actually out of space.
+healthcheck_zfs_capacity_threshold: 80
 healthcheck_ups_name: ""

ansible/roles/healthcheck/tasks/main.yml

@@ -11,10 +11,26 @@
       or healthcheck_command, and a token whenever a push URL is set. A push URL
       with no token would report to Gatus and be rejected 401 on every run.
 
+# Deduplicated across the whole play run. This role is included once PER CHECK,
+# and a host with several checks was otherwise running apt several times to
+# install a curl that was already there - 29 apt transactions estate-wide, and
+# the slowest thing in the deploy by a wide margin. The fact below remembers
+# what has already been ensured on this host.
 - name: Install healthcheck dependencies
   ansible.builtin.package:
-    name: "{{ healthcheck_packages | default(['curl']) }}"
+    name: "{{ healthcheck_wanted_packages }}"
     state: present
+  vars:
+    healthcheck_wanted_packages: >-
+      {{ (healthcheck_packages | default(['curl']))
+         | difference(healthcheck_installed_packages | default([])) }}
+  when: healthcheck_wanted_packages | length > 0
+
+- name: Remember which dependencies this host already has
+  ansible.builtin.set_fact:
+    healthcheck_installed_packages: >-
+      {{ (healthcheck_installed_packages | default([]))
+         | union(healthcheck_packages | default(['curl'])) }}
 
 - name: Create the healthcheck log directory
   ansible.builtin.file:
@@ -49,10 +65,6 @@
     group: root
     mode: "0644"
 
-- name: Reload systemd
-  ansible.builtin.systemd:
-    daemon_reload: yes
-
 # `restarted`, not `started`: started is a no-op on an already-active timer, so
 # a changed interval or a stuck timer would never be picked up.
 - name: "Enable and start the {{ healthcheck_name }} timer"

ansible/roles/healthcheck/templates/checks/zfs-health.sh.j2

@@ -44,9 +44,20 @@
         issues="${issues}${issues:+; }scan errors ${scan_err}"
     fi
 
+    # 6. Capacity. Not an error condition in `zpool status` - a 95% full pool is
+    #    reported perfectly ONLINE - so it has to be read separately, and it is
+    #    the failure you get warning of rather than the one you discover.
+    local capacity
+    capacity=$(zpool list -H -o capacity "$pool" 2>/dev/null | tr -dc '0-9')
+    if [ -z "$capacity" ]; then
+        issues="${issues}${issues:+; }cannot read capacity"
+    elif [ "$capacity" -ge {{ healthcheck_zfs_capacity_threshold }} ]; then
+        issues="${issues}${issues:+; }pool ${capacity}% full (>={{ healthcheck_zfs_capacity_threshold }}%)"
+    fi
+
     if [ -n "$issues" ]; then MESSAGE="$issues"; return 1; fi
 
     local scrub
     scrub=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.start_time // "never"')
-    MESSAGE="${pool} ONLINE, last scrub ${scrub}"
+    MESSAGE="${pool} ONLINE, ${capacity}% full, last scrub ${scrub}"
     return 0