monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity

Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:

  infra/410_disk_usage_alerts.yml   -> disk-usage check   (infra/400)
  infra/420_system_healthcheck.yml  -> liveness check     (infra/400)
  infra/430_cpu_temp_alerts.yml     -> cpu-temp check     (infra/400)
  32_zfs play 2 (monitoring half)   -> zfs-health check   (infra/400)
  34_nut play 2 (entirely)          -> ups-status check   (infra/400)

Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.

Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.

A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.

ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.

Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).

site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.

Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/infra/430_cpu_temp_alerts.yml

@@ -1,324 +0,0 @@
-# ═════════════════════════════════════════════════════════════════════════════
-# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11.
-#
-# This play WILL FAIL if run as-is, and that is deliberate: uptime_kuma_username
-# and uptime_kuma_password were removed from the vault, so the "Validate Uptime
-# Kuma configuration" assert stops it before anything is installed or changed.
-#
-# It is kept because the CHECK LOGIC is the durable part — what gets measured,
-# the thresholds, and the systemd timer plumbing. When something replaces Uptime
-# Kuma, only the push transport needs rewriting; the rest still applies.
-#
-# What was being monitored: archive/uptime_kuma/MONITORS.md
-# ═════════════════════════════════════════════════════════════════════════════
-- name: Deploy CPU Temperature Monitoring
-  hosts: hypervisor
-  become: yes
-
-  vars:
-    temp_threshold_celsius: 80
-    temp_check_interval_minutes: 1
-    monitoring_script_dir: /opt/nodito-monitoring
-    monitoring_script_path: "{{ monitoring_script_dir }}/cpu_temp_monitor.sh"
-    log_file: "{{ monitoring_script_dir }}/cpu_temp_monitor.log"
-    systemd_service_name: nodito-cpu-temp-monitor
-    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
-
-  tasks:
-    - name: Validate Uptime Kuma configuration
-      assert:
-        that:
-          - uptime_kuma_api_url is defined
-          - uptime_kuma_api_url != ""
-          - uptime_kuma_username is defined
-          - uptime_kuma_username != ""
-          - uptime_kuma_password is defined
-          - uptime_kuma_password != ""
-        fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
-
-    - name: Get hostname for monitor identification
-      command: hostname
-      register: host_name
-      changed_when: false
-
-    - name: Set monitor name and group based on hostname
-      set_fact:
-        monitor_name: "cpu-temp-{{ host_name.stdout }}"
-        monitor_friendly_name: "CPU Temperature: {{ host_name.stdout }}"
-        uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
-
-    - name: Create Uptime Kuma CPU temperature monitor setup script
-      copy:
-        dest: /tmp/setup_uptime_kuma_cpu_temp_monitor.py
-        content: |
-          #!/usr/bin/env python3
-          import sys
-          import json
-          from uptime_kuma_api import UptimeKumaApi
-          
-          def main():
-              api_url = sys.argv[1]
-              username = sys.argv[2]
-              password = sys.argv[3]
-              group_name = sys.argv[4]
-              monitor_name = sys.argv[5]
-              monitor_description = sys.argv[6]
-              interval = int(sys.argv[7])
-              ntfy_topic = sys.argv[8] if len(sys.argv) > 8 else "alerts"
-          
-              api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
-              api.login(username, password)
-          
-              monitors = api.get_monitors()
-              notifications = api.get_notifications()
-          
-              ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
-              notification_id_list = {}
-              if ntfy_notification:
-                  notification_id_list[ntfy_notification['id']] = True
-          
-              group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
-              if not group:
-                  api.add_monitor(type='group', name=group_name)
-                  monitors = api.get_monitors()
-                  group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
-          
-              existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
-          
-              monitor_data = {
-                  'type': 'push',
-                  'name': monitor_name,
-                  'parent': group['id'],
-                  'interval': interval,
-                  'upsideDown': True,
-                  'description': monitor_description,
-                  'notificationIDList': notification_id_list
-              }
-          
-              if existing_monitor:
-                  api.edit_monitor(existing_monitor['id'], **monitor_data)
-              else:
-                  api.add_monitor(**monitor_data)
-          
-              monitors = api.get_monitors()
-              monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
-          
-              result = {
-                  'monitor_id': monitor['id'],
-                  'push_token': monitor['pushToken'],
-                  'group_name': group_name,
-                  'group_id': group['id'],
-                  'monitor_name': monitor_name
-              }
-              print(json.dumps(result))
-          
-              api.disconnect()
-          
-          if __name__ == '__main__':
-              main()
-        mode: '0755'
-      delegate_to: localhost
-      become: no
-
-    - name: Run Uptime Kuma monitor setup script
-      command: >
-        {{ ansible_playbook_python }}
-        /tmp/setup_uptime_kuma_cpu_temp_monitor.py
-        "{{ uptime_kuma_api_url }}"
-        "{{ uptime_kuma_username }}"
-        "{{ uptime_kuma_password }}"
-        "{{ uptime_kuma_monitor_group }}"
-        "{{ monitor_name }}"
-        "{{ monitor_friendly_name }} - Alerts when temperature exceeds {{ temp_threshold_celsius }}°C"
-        "{{ (temp_check_interval_minutes * 60) + 60 }}"
-        "{{ ntfy_topic }}"
-      register: monitor_setup_result
-      delegate_to: localhost
-      become: no
-      changed_when: false
-
-    - name: Parse monitor setup result
-      set_fact:
-        monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
-
-    - name: Set push URL and monitor ID as facts
-      set_fact:
-        uptime_kuma_cpu_temp_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
-        uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
-
-    - name: Install required packages for temperature monitoring
-      package:
-        name:
-          - lm-sensors
-          - curl
-          - jq
-          - bc
-        state: present
-
-    - name: Create monitoring script directory
-      file:
-        path: "{{ monitoring_script_dir }}"
-        state: directory
-        owner: root
-        group: root
-        mode: '0755'
-
-    - name: Create CPU temperature monitoring script
-      copy:
-        dest: "{{ monitoring_script_path }}"
-        content: |
-          #!/bin/bash
-          
-          # CPU Temperature Monitoring Script
-          # Monitors CPU temperature and sends alerts to Uptime Kuma
-          
-          LOG_FILE="{{ log_file }}"
-          TEMP_THRESHOLD="{{ temp_threshold_celsius }}"
-          UPTIME_KUMA_URL="{{ uptime_kuma_cpu_temp_push_url }}"
-          
-          log_message() {
-              echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
-          }
-          
-          get_cpu_temp() {
-              local temp=""
-          
-              if command -v sensors >/dev/null 2>&1; then
-                  temp=$(sensors 2>/dev/null | grep -E "Core 0|Package id 0|Tdie|Tctl" | head -1 | grep -oE '[0-9]+\.[0-9]+°C' | grep -oE '[0-9]+\.[0-9]+')
-              fi
-          
-              if [ -z "$temp" ] && [ -f /sys/class/thermal/thermal_zone0/temp ]; then
-                  temp=$(cat /sys/class/thermal/thermal_zone0/temp)
-                  temp=$(echo "scale=1; $temp/1000" | bc -l 2>/dev/null || echo "$temp")
-              fi
-          
-              if [ -z "$temp" ] && command -v acpi >/dev/null 2>&1; then
-                  temp=$(acpi -t 2>/dev/null | grep -oE '[0-9]+\.[0-9]+' | head -1)
-              fi
-          
-              echo "$temp"
-          }
-          
-          send_uptime_kuma_alert() {
-              local temp="$1"
-              local message="CPU Temperature Alert: ${temp}°C (Threshold: ${TEMP_THRESHOLD}°C)"
-          
-              log_message "ALERT: $message"
-          
-              encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/°/%C2%B0/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g')
-              response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
-              http_code=$(echo "$response" | tail -n1)
-          
-              if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
-                  log_message "Alert sent successfully to Uptime Kuma (HTTP $http_code)"
-              else
-                  log_message "ERROR: Failed to send alert to Uptime Kuma (HTTP $http_code)"
-              fi
-          }
-          
-          main() {
-              log_message "Starting CPU temperature check"
-          
-              current_temp=$(get_cpu_temp)
-          
-              if [ -z "$current_temp" ]; then
-                  log_message "ERROR: Could not read CPU temperature"
-                  exit 1
-              fi
-          
-              log_message "Current CPU temperature: ${current_temp}°C"
-          
-              if (( $(echo "$current_temp > $TEMP_THRESHOLD" | bc -l) )); then
-                  log_message "WARNING: CPU temperature ${current_temp}°C exceeds threshold ${TEMP_THRESHOLD}°C"
-                  send_uptime_kuma_alert "$current_temp"
-              else
-                  log_message "CPU temperature is within normal range"
-              fi
-          }
-          
-          main
-        owner: root
-        group: root
-        mode: '0755'
-
-    - name: Create systemd service for CPU temperature monitoring
-      copy:
-        dest: "/etc/systemd/system/{{ systemd_service_name }}.service"
-        content: |
-          [Unit]
-          Description=CPU Temperature Monitor
-          After=network.target
-          
-          [Service]
-          Type=oneshot
-          ExecStart={{ monitoring_script_path }}
-          User=root
-          StandardOutput=journal
-          StandardError=journal
-          
-          [Install]
-          WantedBy=multi-user.target
-        owner: root
-        group: root
-        mode: '0644'
-
-    - name: Create systemd timer for CPU temperature monitoring
-      copy:
-        dest: "/etc/systemd/system/{{ systemd_service_name }}.timer"
-        content: |
-          [Unit]
-          Description=Run CPU Temperature Monitor every {{ temp_check_interval_minutes }} minute(s)
-          Requires={{ systemd_service_name }}.service
-          
-          [Timer]
-          OnBootSec={{ temp_check_interval_minutes }}min
-          OnUnitActiveSec={{ temp_check_interval_minutes }}min
-          Persistent=true
-          
-          [Install]
-          WantedBy=timers.target
-        owner: root
-        group: root
-        mode: '0644'
-
-    - name: Reload systemd daemon
-      systemd:
-        daemon_reload: yes
-
-    - name: Enable and start CPU temperature monitoring timer
-      systemd:
-        name: "{{ systemd_service_name }}.timer"
-        enabled: yes
-        state: started
-
-    - name: Test CPU temperature monitoring script
-      command: "{{ monitoring_script_path }}"
-      register: script_test
-      changed_when: false
-
-    - name: Verify script execution
-      assert:
-        that:
-          - script_test.rc == 0
-        fail_msg: "CPU temperature monitoring script failed to execute properly"
-
-    - name: Display monitoring configuration
-      debug:
-        msg:
-          - "CPU Temperature Monitoring configured successfully"
-          - "Temperature threshold: {{ temp_threshold_celsius }}°C"
-          - "Check interval: {{ temp_check_interval_minutes }} minute(s)"
-          - "Monitor Name: {{ monitor_friendly_name }}"
-          - "Monitor Group: {{ uptime_kuma_monitor_group }}"
-          - "Uptime Kuma Push URL: {{ uptime_kuma_cpu_temp_push_url }}"
-          - "Monitoring script: {{ monitoring_script_path }}"
-          - "Systemd Service: {{ systemd_service_name }}.service"
-          - "Systemd Timer: {{ systemd_service_name }}.timer"
-
-    - name: Clean up temporary Uptime Kuma setup script
-      file:
-        path: /tmp/setup_uptime_kuma_cpu_temp_monitor.py
-        state: absent
-      delegate_to: localhost
-      become: no
-