monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity

Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:

  infra/410_disk_usage_alerts.yml   -> disk-usage check   (infra/400)
  infra/420_system_healthcheck.yml  -> liveness check     (infra/400)
  infra/430_cpu_temp_alerts.yml     -> cpu-temp check     (infra/400)
  32_zfs play 2 (monitoring half)   -> zfs-health check   (infra/400)
  34_nut play 2 (entirely)          -> ups-status check   (infra/400)

Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.

Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.

A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.

ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.

Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).

site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.

Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
counterweight 2026-09-14 10:18:57 +02:00
parent 99760dfd46
commit 853e62a19c
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
16 changed files with 184 additions and 1520 deletions

View file

@ -170,141 +170,55 @@
when: "'ONLINE' not in final_zfs_status.stdout"
# ─────────────────────────────────────────────────────────────────────────────
# ZFS health monitoring and monthly scrub.
# The monthly scrub.
#
# The check script decides healthy/unhealthy and says so in its exit code, which
# systemd keeps: systemctl is-failed zfs-health-monitor.service
# The ZFS HEALTH CHECK that used to share this play is gone: it is now the
# zfs-health check in infra/400_host_monitoring.yml, which carries the same five
# conditions - pool state, device states, resilver in progress, read/write/
# checksum errors, and errors from the last scan - but reports to Gatus like
# every other host check instead of owning its own push plumbing.
#
# Reporting anywhere else is optional. Set `healthcheck_push_url` and the script
# will also GET it with ?status=up|down; leave it empty and the exit code is
# still the whole answer. Today that URL points at Uptime Kuma, which is still
# running on watchtower but is no longer deployed by Ansible - the credentials
# were retired, the service was not. If it is ever replaced, `healthcheck_push_url`
# is the only thing that needs to change here.
# The scrub itself stays here, because it is not monitoring: it is the
# maintenance that gives the health check something true to report. A pool that
# is never scrubbed has no idea whether it is healthy.
# ─────────────────────────────────────────────────────────────────────────────
- name: Setup ZFS Pool Health Monitoring and Monthly Scrubs
- name: Schedule the monthly ZFS scrub
hosts: hypervisor
become: true
vars_files:
- ../../infra_vars.yml
vars:
zfs_check_interval_seconds: 86400 # 24 hours
zfs_check_timeout_seconds: 90000 # ~25 hours (interval + buffer)
zfs_check_retries: 1
zfs_monitoring_script_dir: /opt/zfs-monitoring
zfs_monitoring_script_path: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.sh"
zfs_log_file: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.log"
zfs_systemd_health_service_name: zfs-health-monitor
zfs_systemd_scrub_service_name: zfs-monthly-scrub
# Optional. Empty is fine and is not an error - see the banner above.
healthcheck_push_url: "{{ healthcheck_push_urls.zfs_health | default('') }}"
tasks:
- name: Install required packages for ZFS monitoring
package:
name:
- curl
- jq
state: present
- name: Create monitoring script directory
file:
path: "{{ zfs_monitoring_script_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create ZFS health monitoring script
template:
dest: "{{ zfs_monitoring_script_path }}"
src: templates/zfs_health_monitor.sh.j2
owner: root
group: root
mode: '0755'
- name: Create systemd service for ZFS health monitoring
template:
dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.service"
src: templates/zfs-health-monitor.service.j2
owner: root
group: root
mode: '0644'
- name: Create systemd timer for daily ZFS health monitoring
template:
dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.timer"
src: templates/zfs-health-monitor.timer.j2
owner: root
group: root
mode: '0644'
- name: Create systemd service for ZFS monthly scrub
template:
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.service"
src: templates/zfs-monthly-scrub.service.j2
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.service"
owner: root
group: root
mode: '0644'
- name: Create systemd timer for monthly ZFS scrub
template:
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.timer"
src: templates/zfs-monthly-scrub.timer.j2
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.timer"
owner: root
group: root
mode: '0644'
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start ZFS health monitoring timer
systemd:
name: "{{ zfs_systemd_health_service_name }}.timer"
enabled: yes
state: started
- name: Enable and start ZFS monthly scrub timer
- name: Enable and start the monthly scrub timer
systemd:
name: "{{ zfs_systemd_scrub_service_name }}.timer"
enabled: yes
state: started
daemon_reload: yes
- name: Test ZFS health monitoring script
command: "{{ zfs_monitoring_script_path }}"
register: script_test
changed_when: false
- name: Verify script execution
assert:
that:
- script_test.rc == 0
fail_msg: "ZFS health monitoring script failed - check pool health"
- name: Display monitoring configuration
- name: Report the scrub schedule
debug:
msg: |
✓ ZFS Pool Health Monitoring deployed successfully!
Pool Name: {{ zfs_pool_name }}
Push URL: {{ healthcheck_push_url | default('', true) | ternary('set', 'not set - exit code only') }}
Health Check:
- Frequency: Every {{ zfs_check_interval_seconds }} seconds (24 hours)
- Timeout: {{ zfs_check_timeout_seconds }} seconds (~25 hours)
- Script: {{ zfs_monitoring_script_path }}
- Log: {{ zfs_log_file }}
- Service: {{ zfs_systemd_health_service_name }}.service
- Timer: {{ zfs_systemd_health_service_name }}.timer
Monthly Scrub:
- Schedule: Last day of month at 4:00 AM
- Service: {{ zfs_systemd_scrub_service_name }}.service
- Timer: {{ zfs_systemd_scrub_service_name }}.timer
Conditions monitored:
- Pool state (must be ONLINE)
- Device states (no DEGRADED/FAULTED/OFFLINE/UNAVAIL)
- Resilver status (alerts if resilvering)
- Read/Write/Checksum errors
- Scrub errors
msg: >-
Monthly scrub of {{ zfs_pool_name }}:
last day of each month at 04:00.
Health is reported separately by the zfs-health check
(infra/400_host_monitoring.yml).

View file

@ -225,121 +225,11 @@
- nut-server
- nut-monitor
# ─────────────────────────────────────────────────────────────────────────────
# UPS heartbeat monitoring.
# The UPS heartbeat play that used to live here is gone. What it deployed -
# /opt/ups-monitoring plus a ups-heartbeat timer - is now the ups-status check
# in infra/400_host_monitoring.yml, which reports to Gatus like every other
# host check instead of carrying its own push plumbing.
#
# The script decides on-mains/on-battery and says so in its exit code, which
# systemd keeps: systemctl is-failed ups-heartbeat.service
#
# Reporting anywhere else is optional. Set `healthcheck_push_url` and the script
# will also GET it with ?status=up|down; leave it empty and the exit code is
# still the whole answer. Today that URL points at Uptime Kuma, which is still
# running on watchtower but is no longer deployed by Ansible - the credentials
# were retired, the service was not. If it is ever replaced, `healthcheck_push_url`
# is the only thing that needs to change here.
#
# NOTE: this play has never actually been applied to nodito. /opt/ups-monitoring
# does not exist and there is no ups-heartbeat timer. What is on the box is a
# hand-written /usr/local/bin/ups-heartbeat.sh - mode 0644, not executable, and
# referenced by no unit and no cron entry, so nothing has ever run it. Its push
# token (uLmCPkLLO4) belongs to a monitor that DOES still exist and answer, so
# that monitor has had no heartbeat since January 2026. It is now
# healthcheck_push_urls.ups in the vault, and running this play is what will
# finally start feeding it.
# ─────────────────────────────────────────────────────────────────────────────
- name: Setup UPS Heartbeat Monitoring
hosts: hypervisor
become: true
vars:
ups_heartbeat_interval_seconds: 60
ups_heartbeat_timeout_seconds: 120
ups_heartbeat_retries: 1
ups_monitoring_script_dir: /opt/ups-monitoring
ups_monitoring_script_path: "{{ ups_monitoring_script_dir }}/ups_heartbeat.sh"
ups_log_file: "{{ ups_monitoring_script_dir }}/ups_heartbeat.log"
ups_systemd_service_name: ups-heartbeat
# Optional. Empty is fine and is not an error - see the banner above.
healthcheck_push_url: "{{ healthcheck_push_urls.ups | default('') }}"
tasks:
- name: Install required packages for UPS monitoring
package:
name:
- curl
state: present
- name: Create monitoring script directory
file:
path: "{{ ups_monitoring_script_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create UPS heartbeat monitoring script
template:
dest: "{{ ups_monitoring_script_path }}"
src: templates/ups_heartbeat.sh.j2
owner: root
group: root
mode: '0755'
- name: Create systemd service for UPS heartbeat
template:
dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.service"
src: templates/ups-heartbeat.service.j2
owner: root
group: root
mode: '0644'
- name: Create systemd timer for UPS heartbeat
template:
dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.timer"
src: templates/ups-heartbeat.timer.j2
owner: root
group: root
mode: '0644'
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start UPS heartbeat timer
systemd:
name: "{{ ups_systemd_service_name }}.timer"
enabled: yes
state: started
- name: Test UPS heartbeat script
command: "{{ ups_monitoring_script_path }}"
register: script_test
changed_when: false
- name: Verify script execution
assert:
that:
- script_test.rc == 0
fail_msg: "UPS heartbeat script failed - check UPS status and communication"
- name: Display monitoring configuration
debug:
msg:
- "UPS Monitoring configured successfully"
- ""
- "NUT Configuration:"
- " UPS Name: {{ ups_name }}"
- " UPS Description: {{ ups_desc }}"
- " Off Delay: {{ ups_offdelay }}s (time after shutdown before UPS cuts power)"
- " On Delay: {{ ups_ondelay }}s (time after mains returns before UPS restores power)"
- ""
- "Health reporting:"
- " Check interval: {{ ups_heartbeat_interval_seconds }}s"
- " Push URL: {{ healthcheck_push_url | default('', true) | ternary('set', 'not set - exit code only') }}"
- ""
- "Scripts and Services:"
- " Script: {{ ups_monitoring_script_path }}"
- " Log: {{ ups_log_file }}"
- " Service: {{ ups_systemd_service_name }}.service"
- " Timer: {{ ups_systemd_service_name }}.timer"
# This playbook is now purely NUT setup: the driver, upsd, upsmon and the
# shutdown behaviour. Monitoring whether the UPS is on mains is a separate
# concern and belongs with the other host checks.

View file

@ -1,14 +0,0 @@
[Unit]
Description=UPS Heartbeat Monitor
After=network.target nut-monitor.service
[Service]
Type=oneshot
ExecStart={{ ups_monitoring_script_path }}
User=root
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

View file

@ -1,11 +0,0 @@
[Unit]
Description=Run UPS Heartbeat Monitor every {{ ups_heartbeat_interval_seconds }} seconds
Requires={{ ups_systemd_service_name }}.service
[Timer]
OnBootSec=1min
OnUnitActiveSec={{ ups_heartbeat_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target

View file

@ -1,68 +0,0 @@
#!/bin/bash
# UPS heartbeat check - managed by Ansible (infra/nodito/34_nut_ups_setup_playbook.yml)
#
# The exit code is the answer and systemd keeps it:
# systemctl is-failed {{ ups_systemd_service_name }}.service
# Reporting anywhere else is optional and generic.
LOG_FILE="{{ ups_log_file }}"
UPS_NAME="{{ ups_name }}"
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
report() {
local status="$1"
local message="$2"
# No push URL is normal, not an error: the exit code below is still a
# complete answer for anything reading unit state.
[ -n "$PUSH_URL" ] || return 0
local encoded_message
encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
local response http_code
response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Reported ${status}: $message (HTTP $http_code)"
return 0
else
log_message "ERROR: Failed to report ${status} (HTTP $http_code)"
return 1
fi
}
main() {
local status charge runtime load
status=$(upsc ${UPS_NAME}@localhost ups.status 2>/dev/null)
if [ -z "$status" ]; then
log_message "ERROR: Cannot communicate with UPS"
report "down" "cannot communicate with UPS ${UPS_NAME}"
exit 1
fi
charge=$(upsc ${UPS_NAME}@localhost battery.charge 2>/dev/null)
runtime=$(upsc ${UPS_NAME}@localhost battery.runtime 2>/dev/null)
load=$(upsc ${UPS_NAME}@localhost ups.load 2>/dev/null)
if [[ "$status" == *"OL"* ]]; then
local message="UPS on mains (charge=${charge}% runtime=${runtime}s load=${load}%)"
log_message "$message"
report "up" "$message"
exit 0
else
log_message "UPS not on mains power (status=$status)"
report "down" "UPS not on mains (status=${status} charge=${charge}%)"
exit 1
fi
}
main

View file

@ -1,14 +0,0 @@
[Unit]
Description=ZFS Pool Health Monitor
After=zfs.target network.target
[Service]
Type=oneshot
ExecStart={{ zfs_monitoring_script_path }}
User=root
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

View file

@ -1,11 +0,0 @@
[Unit]
Description=Run ZFS Pool Health Monitor daily
Requires={{ zfs_systemd_health_service_name }}.service
[Timer]
OnBootSec=5min
OnUnitActiveSec={{ zfs_check_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target

View file

@ -1,181 +0,0 @@
#!/bin/bash
# ZFS pool health check - managed by Ansible (infra/nodito/32_zfs_pool_setup_playbook.yml)
#
# The exit code is the answer and systemd keeps it:
# systemctl is-failed {{ zfs_systemd_health_service_name }}.service
# Reporting anywhere else is optional and generic.
LOG_FILE="{{ zfs_log_file }}"
POOL_NAME="{{ zfs_pool_name }}"
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
HOSTNAME=$(hostname)
# Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
# Function to check pool health using JSON output
check_pool_health() {
local pool="$1"
local issues_found=0
# Get pool status as JSON
local pool_json
pool_json=$(zpool status -j "$pool" 2>&1)
if [ $? -ne 0 ]; then
log_message "ERROR: Failed to get pool status for $pool"
log_message " -> $pool_json"
return 1
fi
# Check 1: Pool state must be ONLINE
local pool_state
pool_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].state')
if [ "$pool_state" != "ONLINE" ]; then
log_message "ISSUE: Pool state is $pool_state (expected ONLINE)"
issues_found=1
else
log_message "OK: Pool state is ONLINE"
fi
# Check 2: Check all vdevs and devices for non-ONLINE states
local bad_states
bad_states=$(echo "$pool_json" | jq -r --arg pool "$pool" '
.pools[$pool].vdevs[] |
.. | objects |
select(.state? and .state != "ONLINE") |
"\(.name // "unknown"): \(.state)"
' 2>/dev/null)
if [ -n "$bad_states" ]; then
log_message "ISSUE: Found devices not in ONLINE state:"
echo "$bad_states" | while read -r line; do
log_message " -> $line"
done
issues_found=1
else
log_message "OK: All devices are ONLINE"
fi
# Check 3: Check for resilvering in progress
local scan_function scan_state
scan_function=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
if [ "$scan_function" = "RESILVER" ] && [ "$scan_state" = "SCANNING" ]; then
local resilver_progress
resilver_progress=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.issued // "unknown"')
log_message "ISSUE: Pool is currently resilvering (disk reconstruction in progress) - ${resilver_progress} processed"
issues_found=1
fi
# Check 4: Check for read/write/checksum errors on all devices
# Note: ZFS JSON output has error counts as strings, so convert to numbers for comparison
local devices_with_errors
devices_with_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '
.pools[$pool].vdevs[] |
.. | objects |
select(.name? and ((.read_errors // "0" | tonumber) > 0 or (.write_errors // "0" | tonumber) > 0 or (.checksum_errors // "0" | tonumber) > 0)) |
"\(.name): read=\(.read_errors // 0) write=\(.write_errors // 0) cksum=\(.checksum_errors // 0)"
' 2>/dev/null)
if [ -n "$devices_with_errors" ]; then
log_message "ISSUE: Found devices with I/O errors:"
echo "$devices_with_errors" | while read -r line; do
log_message " -> $line"
done
issues_found=1
else
log_message "OK: No read/write/checksum errors detected"
fi
# Check 5: Check for scan errors (from last scrub/resilver)
local scan_errors
scan_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.errors // "0"')
if [ "$scan_errors" != "0" ] && [ "$scan_errors" != "null" ] && [ -n "$scan_errors" ]; then
log_message "ISSUE: Last scan reported $scan_errors errors"
issues_found=1
else
log_message "OK: No scan errors"
fi
return $issues_found
}
# Function to get last scrub info for status message
get_scrub_info() {
local pool="$1"
local pool_json
pool_json=$(zpool status -j "$pool" 2>/dev/null)
local scan_func scan_state scan_start
scan_func=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
scan_start=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.start_time // ""')
if [ "$scan_func" = "SCRUB" ] && [ "$scan_state" = "SCANNING" ]; then
echo "scrub in progress (started $scan_start)"
elif [ "$scan_func" = "SCRUB" ] && [ -n "$scan_start" ]; then
echo "last scrub: $scan_start"
else
echo "no scrub history"
fi
}
# Optional reporting to whatever is watching. No push URL is normal, not an
# error: the script's exit code is still a complete answer for anything reading
# unit state.
report() {
local status="$1"
local message="$2"
[ -n "$PUSH_URL" ] || return 0
log_message "Reporting ${status}: $message"
# URL encode the message
local encoded_message
encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
local response http_code
response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Report sent successfully (HTTP $http_code)"
return 0
else
log_message "ERROR: Failed to report (HTTP $http_code)"
return 1
fi
}
# Main health check logic
main() {
log_message "=========================================="
log_message "Starting ZFS health check for pool: $POOL_NAME on $HOSTNAME"
# Run all health checks
if check_pool_health "$POOL_NAME"; then
local scrub_info
scrub_info=$(get_scrub_info "$POOL_NAME")
local message="Pool $POOL_NAME healthy ($scrub_info)"
report "up" "$message"
log_message "Health check completed: ALL OK"
exit 0
else
log_message "Health check completed: ISSUES DETECTED"
report "down" "Pool $POOL_NAME unhealthy - see $LOG_FILE"
exit 1
fi
}
# Run main function
main