monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity

Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:

  infra/410_disk_usage_alerts.yml   -> disk-usage check   (infra/400)
  infra/420_system_healthcheck.yml  -> liveness check     (infra/400)
  infra/430_cpu_temp_alerts.yml     -> cpu-temp check     (infra/400)
  32_zfs play 2 (monitoring half)   -> zfs-health check   (infra/400)
  34_nut play 2 (entirely)          -> ups-status check   (infra/400)

Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.

Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.

A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.

ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.

Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).

site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.

Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/infra/nodito/34_nut_ups_setup_playbook.yml

@@ -225,121 +225,11 @@
         - nut-server
         - nut-monitor
 
-
-# ─────────────────────────────────────────────────────────────────────────────
-# UPS heartbeat monitoring.
+# The UPS heartbeat play that used to live here is gone. What it deployed -
+# /opt/ups-monitoring plus a ups-heartbeat timer - is now the ups-status check
+# in infra/400_host_monitoring.yml, which reports to Gatus like every other
+# host check instead of carrying its own push plumbing.
 #
-# The script decides on-mains/on-battery and says so in its exit code, which
-# systemd keeps:  systemctl is-failed ups-heartbeat.service
-#
-# Reporting anywhere else is optional. Set `healthcheck_push_url` and the script
-# will also GET it with ?status=up|down; leave it empty and the exit code is
-# still the whole answer. Today that URL points at Uptime Kuma, which is still
-# running on watchtower but is no longer deployed by Ansible - the credentials
-# were retired, the service was not. If it is ever replaced, `healthcheck_push_url`
-# is the only thing that needs to change here.
-#
-# NOTE: this play has never actually been applied to nodito. /opt/ups-monitoring
-# does not exist and there is no ups-heartbeat timer. What is on the box is a
-# hand-written /usr/local/bin/ups-heartbeat.sh - mode 0644, not executable, and
-# referenced by no unit and no cron entry, so nothing has ever run it. Its push
-# token (uLmCPkLLO4) belongs to a monitor that DOES still exist and answer, so
-# that monitor has had no heartbeat since January 2026. It is now
-# healthcheck_push_urls.ups in the vault, and running this play is what will
-# finally start feeding it.
-# ─────────────────────────────────────────────────────────────────────────────
-- name: Setup UPS Heartbeat Monitoring
-  hosts: hypervisor
-  become: true
-
-  vars:
-    ups_heartbeat_interval_seconds: 60
-    ups_heartbeat_timeout_seconds: 120
-    ups_heartbeat_retries: 1
-    ups_monitoring_script_dir: /opt/ups-monitoring
-    ups_monitoring_script_path: "{{ ups_monitoring_script_dir }}/ups_heartbeat.sh"
-    ups_log_file: "{{ ups_monitoring_script_dir }}/ups_heartbeat.log"
-    ups_systemd_service_name: ups-heartbeat
-    # Optional. Empty is fine and is not an error - see the banner above.
-    healthcheck_push_url: "{{ healthcheck_push_urls.ups | default('') }}"
-
-  tasks:
-    - name: Install required packages for UPS monitoring
-      package:
-        name:
-          - curl
-        state: present
-
-    - name: Create monitoring script directory
-      file:
-        path: "{{ ups_monitoring_script_dir }}"
-        state: directory
-        owner: root
-        group: root
-        mode: '0755'
-
-    - name: Create UPS heartbeat monitoring script
-      template:
-        dest: "{{ ups_monitoring_script_path }}"
-        src: templates/ups_heartbeat.sh.j2
-        owner: root
-        group: root
-        mode: '0755'
-
-    - name: Create systemd service for UPS heartbeat
-      template:
-        dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.service"
-        src: templates/ups-heartbeat.service.j2
-        owner: root
-        group: root
-        mode: '0644'
-
-    - name: Create systemd timer for UPS heartbeat
-      template:
-        dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.timer"
-        src: templates/ups-heartbeat.timer.j2
-        owner: root
-        group: root
-        mode: '0644'
-
-    - name: Reload systemd daemon
-      systemd:
-        daemon_reload: yes
-
-    - name: Enable and start UPS heartbeat timer
-      systemd:
-        name: "{{ ups_systemd_service_name }}.timer"
-        enabled: yes
-        state: started
-
-    - name: Test UPS heartbeat script
-      command: "{{ ups_monitoring_script_path }}"
-      register: script_test
-      changed_when: false
-
-    - name: Verify script execution
-      assert:
-        that:
-          - script_test.rc == 0
-        fail_msg: "UPS heartbeat script failed - check UPS status and communication"
-
-    - name: Display monitoring configuration
-      debug:
-        msg:
-          - "UPS Monitoring configured successfully"
-          - ""
-          - "NUT Configuration:"
-          - "  UPS Name: {{ ups_name }}"
-          - "  UPS Description: {{ ups_desc }}"
-          - "  Off Delay: {{ ups_offdelay }}s (time after shutdown before UPS cuts power)"
-          - "  On Delay: {{ ups_ondelay }}s (time after mains returns before UPS restores power)"
-          - ""
-          - "Health reporting:"
-          - "  Check interval: {{ ups_heartbeat_interval_seconds }}s"
-          - "  Push URL: {{ healthcheck_push_url | default('', true) | ternary('set', 'not set - exit code only') }}"
-          - ""
-          - "Scripts and Services:"
-          - "  Script: {{ ups_monitoring_script_path }}"
-          - "  Log: {{ ups_log_file }}"
-          - "  Service: {{ ups_systemd_service_name }}.service"
-          - "  Timer: {{ ups_systemd_service_name }}.timer"
+# This playbook is now purely NUT setup: the driver, upsd, upsmon and the
+# shutdown behaviour. Monitoring whether the UPS is on mains is a separate
+# concern and belongs with the other host checks.