monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity
Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
99760dfd46
commit
853e62a19c
GPG key ID:
883EDBAA726BD96C
16 changed files with 184 additions and 1520 deletions
|
|
@ -1,14 +0,0 @@
|
|||
[Unit]
|
||||
Description=UPS Heartbeat Monitor
|
||||
After=network.target nut-monitor.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart={{ ups_monitoring_script_path }}
|
||||
User=root
|
||||
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
[Unit]
|
||||
Description=Run UPS Heartbeat Monitor every {{ ups_heartbeat_interval_seconds }} seconds
|
||||
Requires={{ ups_systemd_service_name }}.service
|
||||
|
||||
[Timer]
|
||||
OnBootSec=1min
|
||||
OnUnitActiveSec={{ ups_heartbeat_interval_seconds }}sec
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# UPS heartbeat check - managed by Ansible (infra/nodito/34_nut_ups_setup_playbook.yml)
|
||||
#
|
||||
# The exit code is the answer and systemd keeps it:
|
||||
# systemctl is-failed {{ ups_systemd_service_name }}.service
|
||||
# Reporting anywhere else is optional and generic.
|
||||
|
||||
LOG_FILE="{{ ups_log_file }}"
|
||||
UPS_NAME="{{ ups_name }}"
|
||||
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
|
||||
|
||||
log_message() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
report() {
|
||||
local status="$1"
|
||||
local message="$2"
|
||||
|
||||
# No push URL is normal, not an error: the exit code below is still a
|
||||
# complete answer for anything reading unit state.
|
||||
[ -n "$PUSH_URL" ] || return 0
|
||||
|
||||
local encoded_message
|
||||
encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
|
||||
|
||||
local response http_code
|
||||
response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
|
||||
http_code=$(echo "$response" | tail -n1)
|
||||
|
||||
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
|
||||
log_message "Reported ${status}: $message (HTTP $http_code)"
|
||||
return 0
|
||||
else
|
||||
log_message "ERROR: Failed to report ${status} (HTTP $http_code)"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
local status charge runtime load
|
||||
|
||||
status=$(upsc ${UPS_NAME}@localhost ups.status 2>/dev/null)
|
||||
|
||||
if [ -z "$status" ]; then
|
||||
log_message "ERROR: Cannot communicate with UPS"
|
||||
report "down" "cannot communicate with UPS ${UPS_NAME}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
charge=$(upsc ${UPS_NAME}@localhost battery.charge 2>/dev/null)
|
||||
runtime=$(upsc ${UPS_NAME}@localhost battery.runtime 2>/dev/null)
|
||||
load=$(upsc ${UPS_NAME}@localhost ups.load 2>/dev/null)
|
||||
|
||||
if [[ "$status" == *"OL"* ]]; then
|
||||
local message="UPS on mains (charge=${charge}% runtime=${runtime}s load=${load}%)"
|
||||
log_message "$message"
|
||||
report "up" "$message"
|
||||
exit 0
|
||||
else
|
||||
log_message "UPS not on mains power (status=$status)"
|
||||
report "down" "UPS not on mains (status=${status} charge=${charge}%)"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
main
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
[Unit]
|
||||
Description=ZFS Pool Health Monitor
|
||||
After=zfs.target network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart={{ zfs_monitoring_script_path }}
|
||||
User=root
|
||||
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
[Unit]
|
||||
Description=Run ZFS Pool Health Monitor daily
|
||||
Requires={{ zfs_systemd_health_service_name }}.service
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
OnUnitActiveSec={{ zfs_check_interval_seconds }}sec
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
|
|
@ -1,181 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# ZFS pool health check - managed by Ansible (infra/nodito/32_zfs_pool_setup_playbook.yml)
|
||||
#
|
||||
# The exit code is the answer and systemd keeps it:
|
||||
# systemctl is-failed {{ zfs_systemd_health_service_name }}.service
|
||||
# Reporting anywhere else is optional and generic.
|
||||
|
||||
LOG_FILE="{{ zfs_log_file }}"
|
||||
POOL_NAME="{{ zfs_pool_name }}"
|
||||
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
|
||||
HOSTNAME=$(hostname)
|
||||
|
||||
# Function to log messages
|
||||
log_message() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
# Function to check pool health using JSON output
|
||||
check_pool_health() {
|
||||
local pool="$1"
|
||||
local issues_found=0
|
||||
|
||||
# Get pool status as JSON
|
||||
local pool_json
|
||||
pool_json=$(zpool status -j "$pool" 2>&1)
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
log_message "ERROR: Failed to get pool status for $pool"
|
||||
log_message " -> $pool_json"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Check 1: Pool state must be ONLINE
|
||||
local pool_state
|
||||
pool_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].state')
|
||||
|
||||
if [ "$pool_state" != "ONLINE" ]; then
|
||||
log_message "ISSUE: Pool state is $pool_state (expected ONLINE)"
|
||||
issues_found=1
|
||||
else
|
||||
log_message "OK: Pool state is ONLINE"
|
||||
fi
|
||||
|
||||
# Check 2: Check all vdevs and devices for non-ONLINE states
|
||||
local bad_states
|
||||
bad_states=$(echo "$pool_json" | jq -r --arg pool "$pool" '
|
||||
.pools[$pool].vdevs[] |
|
||||
.. | objects |
|
||||
select(.state? and .state != "ONLINE") |
|
||||
"\(.name // "unknown"): \(.state)"
|
||||
' 2>/dev/null)
|
||||
|
||||
if [ -n "$bad_states" ]; then
|
||||
log_message "ISSUE: Found devices not in ONLINE state:"
|
||||
echo "$bad_states" | while read -r line; do
|
||||
log_message " -> $line"
|
||||
done
|
||||
issues_found=1
|
||||
else
|
||||
log_message "OK: All devices are ONLINE"
|
||||
fi
|
||||
|
||||
# Check 3: Check for resilvering in progress
|
||||
local scan_function scan_state
|
||||
scan_function=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
|
||||
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
|
||||
|
||||
if [ "$scan_function" = "RESILVER" ] && [ "$scan_state" = "SCANNING" ]; then
|
||||
local resilver_progress
|
||||
resilver_progress=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.issued // "unknown"')
|
||||
log_message "ISSUE: Pool is currently resilvering (disk reconstruction in progress) - ${resilver_progress} processed"
|
||||
issues_found=1
|
||||
fi
|
||||
|
||||
# Check 4: Check for read/write/checksum errors on all devices
|
||||
# Note: ZFS JSON output has error counts as strings, so convert to numbers for comparison
|
||||
local devices_with_errors
|
||||
devices_with_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '
|
||||
.pools[$pool].vdevs[] |
|
||||
.. | objects |
|
||||
select(.name? and ((.read_errors // "0" | tonumber) > 0 or (.write_errors // "0" | tonumber) > 0 or (.checksum_errors // "0" | tonumber) > 0)) |
|
||||
"\(.name): read=\(.read_errors // 0) write=\(.write_errors // 0) cksum=\(.checksum_errors // 0)"
|
||||
' 2>/dev/null)
|
||||
|
||||
if [ -n "$devices_with_errors" ]; then
|
||||
log_message "ISSUE: Found devices with I/O errors:"
|
||||
echo "$devices_with_errors" | while read -r line; do
|
||||
log_message " -> $line"
|
||||
done
|
||||
issues_found=1
|
||||
else
|
||||
log_message "OK: No read/write/checksum errors detected"
|
||||
fi
|
||||
|
||||
# Check 5: Check for scan errors (from last scrub/resilver)
|
||||
local scan_errors
|
||||
scan_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.errors // "0"')
|
||||
|
||||
if [ "$scan_errors" != "0" ] && [ "$scan_errors" != "null" ] && [ -n "$scan_errors" ]; then
|
||||
log_message "ISSUE: Last scan reported $scan_errors errors"
|
||||
issues_found=1
|
||||
else
|
||||
log_message "OK: No scan errors"
|
||||
fi
|
||||
|
||||
return $issues_found
|
||||
}
|
||||
|
||||
# Function to get last scrub info for status message
|
||||
get_scrub_info() {
|
||||
local pool="$1"
|
||||
local pool_json
|
||||
pool_json=$(zpool status -j "$pool" 2>/dev/null)
|
||||
|
||||
local scan_func scan_state scan_start
|
||||
scan_func=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
|
||||
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
|
||||
scan_start=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.start_time // ""')
|
||||
|
||||
if [ "$scan_func" = "SCRUB" ] && [ "$scan_state" = "SCANNING" ]; then
|
||||
echo "scrub in progress (started $scan_start)"
|
||||
elif [ "$scan_func" = "SCRUB" ] && [ -n "$scan_start" ]; then
|
||||
echo "last scrub: $scan_start"
|
||||
else
|
||||
echo "no scrub history"
|
||||
fi
|
||||
}
|
||||
|
||||
# Optional reporting to whatever is watching. No push URL is normal, not an
|
||||
# error: the script's exit code is still a complete answer for anything reading
|
||||
# unit state.
|
||||
report() {
|
||||
local status="$1"
|
||||
local message="$2"
|
||||
|
||||
[ -n "$PUSH_URL" ] || return 0
|
||||
|
||||
log_message "Reporting ${status}: $message"
|
||||
|
||||
# URL encode the message
|
||||
local encoded_message
|
||||
encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
|
||||
|
||||
local response http_code
|
||||
response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
|
||||
http_code=$(echo "$response" | tail -n1)
|
||||
|
||||
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
|
||||
log_message "Report sent successfully (HTTP $http_code)"
|
||||
return 0
|
||||
else
|
||||
log_message "ERROR: Failed to report (HTTP $http_code)"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Main health check logic
|
||||
main() {
|
||||
log_message "=========================================="
|
||||
log_message "Starting ZFS health check for pool: $POOL_NAME on $HOSTNAME"
|
||||
|
||||
# Run all health checks
|
||||
if check_pool_health "$POOL_NAME"; then
|
||||
local scrub_info
|
||||
scrub_info=$(get_scrub_info "$POOL_NAME")
|
||||
|
||||
local message="Pool $POOL_NAME healthy ($scrub_info)"
|
||||
report "up" "$message"
|
||||
|
||||
log_message "Health check completed: ALL OK"
|
||||
exit 0
|
||||
else
|
||||
log_message "Health check completed: ISSUES DETECTED"
|
||||
report "down" "Pool $POOL_NAME unhealthy - see $LOG_FILE"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Run main function
|
||||
main
|
||||
Loading…
Add table
Add a link
Reference in a new issue