counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

nodito: de-Uptime-Kuma the ZFS and NUT playbooks, extract templates

Browse source 
These two are host-specific by design - nodito is a pet, not cattle - so they
stay playbooks rather than becoming roles. But they had rotted.

De-Kuma, following the pattern of the six service roles:

  Both plays opened with an assert on uptime_kuma_username/password, which were
  removed from the vault, so both failed before doing anything. Dropped that,
  the two embedded Python monitor-creation scripts, and their /tmp cleanup.
  Kept every check, threshold and systemd timer - those are the durable part.

  Reporting is now generic: `healthcheck_push_url` goes into the unit as
  Environment=HEALTHCHECK_PUSH_URL and the script reads ${HEALTHCHECK_PUSH_URL:-},
  treating empty as normal rather than an error. The exit code is the real
  answer; systemd keeps it. Both scripts now also report status=down on failure
  instead of only going silent. Live push URLs harvested into the vault so
  nothing observable changes for ZFS.

Three live bugs found while check-diffing:

  1. 32_zfs would have DE-REGISTERED the Proxmox storage. `pvesm remove` was
     gated on the storage existing and `pvesm add` on it NOT existing - mutually
     exclusive - so a real run removed the proxmox-tank-1 entry backing every VM
     and never put it back. It would also have dropped `mountpoint /var/lib/vz`,
     which the live entry has and `pvesm add` does not set. Registration is now
     add-only.

  2. zfs_disk_1 named ata-...WX11TN0Z, a disk no longer in the machine. The live
     mirror is WX120LHQ + WX11TN2P; a leg was replaced and the repo never caught
     up. Inert behind `when: zfs_pool_exists.rc != 0`, but wrong on any
     disaster-recovery run. This is the seventh instance of an identifier
     written down once whose hardware later moved.

  3. 34_nut has NEVER been applied to nodito - no /etc/nut file carries the
     "Managed by Ansible" marker; they were written by hand in January 2026. The
     vault held the literal CHANGE_ME_TO_SECURE_PASSWORD, so applying it would
     have overwritten a working upsd/upsmon auth pair with a placeholder and
     restarted NUT, leaving the hypervisor's UPS unable to trigger a clean
     shutdown on mains loss. The Kuma assert was the only thing stopping that,
     so removing it without a replacement would have armed the gun: there is now
     an explicit assert that refuses to run on the placeholder. The real
     password is in the vault and `Configure upsd users` check-diffs clean.

  Templates reconciled with the live files first, so applying 34_nut is close to
  a no-op: added `maxretry = 3` to ups.conf and OFFDURATION / RBWARNTIME /
  NOCOMMWARNTIME / FINALDELAY plus quoted POWERDOWNFLAG to upsmon.conf. The only
  substantive additions left are the NOTIFYMSG/NOTIFYFLAG syslog lines.

  /usr/local/bin/ups-heartbeat.sh on the box is an orphan - mode 0644, not
  executable, referenced by no unit and no cron entry - but its push token
  belongs to a monitor that still exists and answers, so that monitor has had no
  heartbeat since January. Harvested as healthcheck_push_urls.ups; applying this
  play is what will finally feed it.

Finish the host_vars migration:

  infra/nodito/nodito_vars.yml was byte-identical to host_vars/nodito/main.yml.
  Deleted it, moved nodito_secrets.yml to host_vars/nodito/vault.yml, and
  stripped the dead vars_files entries from all three playbooks.

Extract the 13 inline `content: |` blocks to infra/nodito/templates/, pulled via
a YAML load rather than retyped. 681+582 lines become 313+344 plus templates.
Ownership parity against HEAD checked mechanically: no owner/group/mode drift on
any surviving task.

Neither playbook has been applied. ZFS play 2 check-runs failed=0 with two
changes: the script rewrite (the deployed one is a hand-edited DEBUG VERSION
with `set -x`) and the Environment line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
counterweight 2026-09-13 19:07:50 +02:00
parent a0c23ae766
commit 0f03c503c8
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
22 changed files with 876 additions and 1122 deletions
Download patch file Download diff file Expand all files Collapse all files

435
ansible/infra/nodito/32_zfs_pool_setup_playbook.yml
View file

@ -3,7 +3,6 @@
become: true
vars_files:
- ../infra_vars.yml
- nodito_vars.yml
tasks:
- name: Verify Proxmox VE is running
@ -139,17 +138,18 @@
Config file exists: {{ storage_cfg_file.stat.exists }}
Storage check result: {{ storage_exists_check.rc }}
Pool exists: {{ zfs_pool_exists.rc == 0 }}
Will remove storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc == 0 }}
Will add storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc != 0 }}
- name: Remove existing storage if it exists
command: pvesm remove {{ zfs_pool_name }}
register: pvesm_remove_result
failed_when: false
when:
- zfs_pool_exists.rc == 0
- storage_exists_check.rc == 0
# Registration is add-only on purpose. There used to be a "Remove existing
# storage if it exists" task here that ran `pvesm remove` whenever the
# storage WAS present, paired with an add that only ran when it was ABSENT.
# The two conditions are mutually exclusive, so a real run against a
# correctly-configured hypervisor removed the storage entry backing every VM
# and never put it back. It would also have dropped `mountpoint /var/lib/vz`,
# which the live entry has and which `pvesm add` below does not set.
#
# If the storage entry ever needs its options changed, edit
# /etc/pve/storage.cfg or use `pvesm set` - do not re-register it from here.
- name: Add ZFS pool storage to Proxmox using pvesm
command: >
pvesm add zfspool {{ zfs_pool_name }}
@ -171,27 +171,24 @@
msg: "ZFS pool {{ zfs_pool_name }} is not in a healthy state"
when: "'ONLINE' not in final_zfs_status.stdout"
# ═════════════════════════════════════════════════════════════════════════════
# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11.
# ─────────────────────────────────────────────────────────────────────────────
# ZFS health monitoring and monthly scrub.
#
# This play WILL FAIL if run as-is, and that is deliberate: uptime_kuma_username
# and uptime_kuma_password were removed from the vault, so the "Validate Uptime
# Kuma configuration" assert stops it before anything is installed or changed.
# The check script decides healthy/unhealthy and says so in its exit code, which
# systemd keeps: systemctl is-failed zfs-health-monitor.service
#
# It is kept because the CHECK LOGIC is the durable part — what gets measured,
# the thresholds, and the systemd timer plumbing. When something replaces Uptime
# Kuma, only the push transport needs rewriting; the rest still applies.
#
# What was being monitored: archive/uptime_kuma/MONITORS.md
# ═════════════════════════════════════════════════════════════════════════════
# Reporting anywhere else is optional. Set `healthcheck_push_url` and the script
# will also GET it with ?status=up|down; leave it empty and the exit code is
# still the whole answer. Today that URL points at Uptime Kuma, which is still
# running on watchtower but is no longer deployed by Ansible - the credentials
# were retired, the service was not. If it is ever replaced, `healthcheck_push_url`
# is the only thing that needs to change here.
# ─────────────────────────────────────────────────────────────────────────────
- name: Setup ZFS Pool Health Monitoring and Monthly Scrubs
hosts: hypervisor
become: true
vars_files:
- ../../infra_vars.yml
- ../../services_config.yml
- ../../infra_secrets.yml
- nodito_vars.yml
vars:
zfs_check_interval_seconds: 86400 # 24 hours
@ -202,143 +199,10 @@
zfs_log_file: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.log"
zfs_systemd_health_service_name: zfs-health-monitor
zfs_systemd_scrub_service_name: zfs-monthly-scrub
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
ntfy_topic: "{{ service_settings.ntfy.topic }}"
# Optional. Empty is fine and is not an error - see the banner above.
healthcheck_push_url: "{{ healthcheck_push_urls.zfs_health | default('') }}"
tasks:
- name: Validate Uptime Kuma configuration
assert:
that:
- uptime_kuma_api_url is defined
- uptime_kuma_api_url != ""
- uptime_kuma_username is defined
- uptime_kuma_username != ""
- uptime_kuma_password is defined
- uptime_kuma_password != ""
fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
- name: Get hostname for monitor identification
command: hostname
register: host_name
changed_when: false
- name: Set monitor name and group based on hostname
set_fact:
monitor_name: "zfs-health-{{ host_name.stdout }}"
monitor_friendly_name: "ZFS Pool Health: {{ host_name.stdout }}"
uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
- name: Create Uptime Kuma ZFS health monitor setup script
copy:
dest: /tmp/setup_uptime_kuma_zfs_monitor.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
group_name = sys.argv[4]
monitor_name = sys.argv[5]
monitor_description = sys.argv[6]
interval = int(sys.argv[7])
retries = int(sys.argv[8])
ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts"
api = UptimeKumaApi(api_url, timeout=120, wait_events=2.0)
api.login(username, password)
# Get all monitors
monitors = api.get_monitors()
# Get all notifications and find ntfy notification
notifications = api.get_notifications()
ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
notification_id_list = {}
if ntfy_notification:
notification_id_list[ntfy_notification['id']] = True
# Find or create group
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
if not group:
group_result = api.add_monitor(type='group', name=group_name)
# Refresh to get the full group object with id
monitors = api.get_monitors()
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
# Find or create/update push monitor
existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
monitor_data = {
'type': 'push',
'name': monitor_name,
'parent': group['id'],
'interval': interval,
'upsideDown': False, # Normal heartbeat mode: receiving pings = healthy
'maxretries': retries,
'description': monitor_description,
'notificationIDList': notification_id_list
}
if existing_monitor:
monitor = api.edit_monitor(existing_monitor['id'], **monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
else:
monitor_result = api.add_monitor(**monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
# Output result as JSON
result = {
'monitor_id': monitor['id'],
'push_token': monitor['pushToken'],
'group_name': group_name,
'group_id': group['id'],
'monitor_name': monitor_name
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma ZFS monitor setup script
command: >
{{ ansible_playbook_python }}
/tmp/setup_uptime_kuma_zfs_monitor.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"{{ uptime_kuma_monitor_group }}"
"{{ monitor_name }}"
"{{ monitor_friendly_name }} - Daily health check for pool {{ zfs_pool_name }}"
"{{ zfs_check_timeout_seconds }}"
"{{ zfs_check_retries }}"
"{{ ntfy_topic }}"
register: monitor_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse monitor setup result
set_fact:
monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
- name: Set push URL and monitor ID as facts
set_fact:
uptime_kuma_zfs_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
- name: Install required packages for ZFS monitoring
package:
name:
@ -355,264 +219,41 @@
mode: '0755'
- name: Create ZFS health monitoring script
copy:
template:
dest: "{{ zfs_monitoring_script_path }}"
content: |
#!/bin/bash
# ZFS Pool Health Monitoring Script
# Checks ZFS pool health using JSON output and sends heartbeat to Uptime Kuma if healthy
# If any issues detected, does NOT send heartbeat (triggers timeout alert)
LOG_FILE="{{ zfs_log_file }}"
UPTIME_KUMA_URL="{{ uptime_kuma_zfs_push_url }}"
POOL_NAME="{{ zfs_pool_name }}"
HOSTNAME=$(hostname)
# Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
# Function to check pool health using JSON output
check_pool_health() {
local pool="$1"
local issues_found=0
# Get pool status as JSON
local pool_json
pool_json=$(zpool status -j "$pool" 2>&1)
if [ $? -ne 0 ]; then
log_message "ERROR: Failed to get pool status for $pool"
log_message " -> $pool_json"
return 1
fi
# Check 1: Pool state must be ONLINE
local pool_state
pool_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].state')
if [ "$pool_state" != "ONLINE" ]; then
log_message "ISSUE: Pool state is $pool_state (expected ONLINE)"
issues_found=1
else
log_message "OK: Pool state is ONLINE"
fi
# Check 2: Check all vdevs and devices for non-ONLINE states
local bad_states
bad_states=$(echo "$pool_json" | jq -r --arg pool "$pool" '
.pools[$pool].vdevs[] |
.. | objects |
select(.state? and .state != "ONLINE") |
"\(.name // "unknown"): \(.state)"
' 2>/dev/null)
if [ -n "$bad_states" ]; then
log_message "ISSUE: Found devices not in ONLINE state:"
echo "$bad_states" | while read -r line; do
log_message " -> $line"
done
issues_found=1
else
log_message "OK: All devices are ONLINE"
fi
# Check 3: Check for resilvering in progress
local scan_function scan_state
scan_function=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
if [ "$scan_function" = "RESILVER" ] && [ "$scan_state" = "SCANNING" ]; then
local resilver_progress
resilver_progress=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.issued // "unknown"')
log_message "ISSUE: Pool is currently resilvering (disk reconstruction in progress) - ${resilver_progress} processed"
issues_found=1
fi
# Check 4: Check for read/write/checksum errors on all devices
# Note: ZFS JSON output has error counts as strings, so convert to numbers for comparison
local devices_with_errors
devices_with_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '
.pools[$pool].vdevs[] |
.. | objects |
select(.name? and ((.read_errors // "0" | tonumber) > 0 or (.write_errors // "0" | tonumber) > 0 or (.checksum_errors // "0" | tonumber) > 0)) |
"\(.name): read=\(.read_errors // 0) write=\(.write_errors // 0) cksum=\(.checksum_errors // 0)"
' 2>/dev/null)
if [ -n "$devices_with_errors" ]; then
log_message "ISSUE: Found devices with I/O errors:"
echo "$devices_with_errors" | while read -r line; do
log_message " -> $line"
done
issues_found=1
else
log_message "OK: No read/write/checksum errors detected"
fi
# Check 5: Check for scan errors (from last scrub/resilver)
local scan_errors
scan_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.errors // "0"')
if [ "$scan_errors" != "0" ] && [ "$scan_errors" != "null" ] && [ -n "$scan_errors" ]; then
log_message "ISSUE: Last scan reported $scan_errors errors"
issues_found=1
else
log_message "OK: No scan errors"
fi
return $issues_found
}
# Function to get last scrub info for status message
get_scrub_info() {
local pool="$1"
local pool_json
pool_json=$(zpool status -j "$pool" 2>/dev/null)
local scan_func scan_state scan_start
scan_func=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
scan_start=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.start_time // ""')
if [ "$scan_func" = "SCRUB" ] && [ "$scan_state" = "SCANNING" ]; then
echo "scrub in progress (started $scan_start)"
elif [ "$scan_func" = "SCRUB" ] && [ -n "$scan_start" ]; then
echo "last scrub: $scan_start"
else
echo "no scrub history"
fi
}
# Function to send heartbeat to Uptime Kuma
send_heartbeat() {
local message="$1"
log_message "Sending heartbeat to Uptime Kuma: $message"
# URL encode the message
local encoded_message
encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
local response http_code
response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Heartbeat sent successfully (HTTP $http_code)"
return 0
else
log_message "ERROR: Failed to send heartbeat (HTTP $http_code)"
return 1
fi
}
# Main health check logic
main() {
log_message "=========================================="
log_message "Starting ZFS health check for pool: $POOL_NAME on $HOSTNAME"
# Run all health checks
if check_pool_health "$POOL_NAME"; then
# All checks passed - send heartbeat
local scrub_info
scrub_info=$(get_scrub_info "$POOL_NAME")
local message="Pool $POOL_NAME healthy ($scrub_info)"
send_heartbeat "$message"
log_message "Health check completed: ALL OK"
exit 0
else
# Issues found - do NOT send heartbeat (will trigger timeout alert)
log_message "Health check completed: ISSUES DETECTED - NOT sending heartbeat"
log_message "Uptime Kuma will alert after timeout due to missing heartbeat"
exit 1
fi
}
# Run main function
main
src: templates/zfs_health_monitor.sh.j2
owner: root
group: root
mode: '0755'
- name: Create systemd service for ZFS health monitoring
copy:
template:
dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.service"
content: |
[Unit]
Description=ZFS Pool Health Monitor
After=zfs.target network.target
[Service]
Type=oneshot
ExecStart={{ zfs_monitoring_script_path }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
src: templates/zfs-health-monitor.service.j2
owner: root
group: root
mode: '0644'
- name: Create systemd timer for daily ZFS health monitoring
copy:
template:
dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.timer"
content: |
[Unit]
Description=Run ZFS Pool Health Monitor daily
Requires={{ zfs_systemd_health_service_name }}.service
[Timer]
OnBootSec=5min
OnUnitActiveSec={{ zfs_check_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target
src: templates/zfs-health-monitor.timer.j2
owner: root
group: root
mode: '0644'
- name: Create systemd service for ZFS monthly scrub
copy:
template:
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.service"
content: |
[Unit]
Description=ZFS Monthly Scrub for {{ zfs_pool_name }}
After=zfs.target
[Service]
Type=oneshot
ExecStart=/sbin/zpool scrub {{ zfs_pool_name }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
src: templates/zfs-monthly-scrub.service.j2
owner: root
group: root
mode: '0644'
- name: Create systemd timer for monthly ZFS scrub
copy:
template:
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.timer"
content: |
[Unit]
Description=Run ZFS Scrub on last day of every month at 4:00 AM
Requires={{ zfs_systemd_scrub_service_name }}.service
[Timer]
OnCalendar=*-*~01 04:00:00
Persistent=true
[Install]
WantedBy=timers.target
src: templates/zfs-monthly-scrub.timer.j2
owner: root
group: root
mode: '0644'
@ -649,9 +290,8 @@
msg: |
✓ ZFS Pool Health Monitoring deployed successfully!
Monitor Name: {{ monitor_friendly_name }}
Monitor Group: {{ uptime_kuma_monitor_group }}
Pool Name: {{ zfs_pool_name }}
Push URL: {{ healthcheck_push_url | default('', true) | ternary('set', 'not set - exit code only') }}
Health Check:
- Frequency: Every {{ zfs_check_interval_seconds }} seconds (24 hours)
@ -672,10 +312,3 @@
- Resilver status (alerts if resilvering)
- Read/Write/Checksum errors
- Scrub errors
- name: Clean up temporary Uptime Kuma setup script
file:
path: /tmp/setup_uptime_kuma_zfs_monitor.py
state: absent
delegate_to: localhost
become: no

1
ansible/infra/nodito/33_proxmox_debian_cloud_template.yml
View file

@ -3,7 +3,6 @@
become: true
vars_files:
- ../../infra_vars.yml
- nodito_vars.yml
vars:
# Defaults (override via vars_files or --extra-vars as needed)

365
ansible/infra/nodito/34_nut_ups_setup_playbook.yml
View file

@ -3,10 +3,33 @@
become: true
vars_files:
- ../../infra_vars.yml
- nodito_vars.yml
- nodito_secrets.yml
tasks:
# ------------------------------------------------------------------
# Safety catch
#
# /etc/nut/upsd.users and /etc/nut/upsmon.conf on nodito were written by
# hand in January 2026 and carry a working password. host_vars/nodito/vault.yml
# (formerly infra/nodito/nodito_secrets.yml) still holds the literal string
# CHANGE_ME_TO_SECURE_PASSWORD, so running this play would overwrite that
# working pair with a placeholder and restart NUT - leaving the hypervisor's
# UPS unmonitored and unable to trigger a clean shutdown on mains loss.
#
# Until the real password is put in the vault, stop here.
# ansible-vault edit host_vars/nodito/vault.yml
# ------------------------------------------------------------------
- name: Refuse to run with a placeholder UPS password
assert:
that:
- ups_password is defined
- ups_password | length > 0
- ups_password != "CHANGE_ME_TO_SECURE_PASSWORD"
fail_msg: >-
ups_password is unset or still the placeholder. Applying this play would
overwrite the working /etc/nut/upsd.users and /etc/nut/upsmon.conf on
nodito and restart NUT. Put the real password in the vault first:
ansible-vault edit host_vars/nodito/vault.yml
# ------------------------------------------------------------------
# Installation
# ------------------------------------------------------------------
@ -75,90 +98,45 @@
# Configuration files
# ------------------------------------------------------------------
- name: Configure NUT mode (standalone)
copy:
template:
dest: /etc/nut/nut.conf
content: |
# Managed by Ansible
MODE=standalone
src: templates/nut.conf.j2
owner: root
group: nut
mode: "0640"
notify: Restart NUT services
- name: Configure UPS device
copy:
template:
dest: /etc/nut/ups.conf
content: |
# Managed by Ansible
[{{ ups_name }}]
driver = {{ ups_driver }}
port = {{ ups_port }}
desc = "{{ ups_desc }}"
offdelay = {{ ups_offdelay }}
ondelay = {{ ups_ondelay }}
src: templates/ups.conf.j2
owner: root
group: nut
mode: "0640"
notify: Restart NUT services
- name: Configure upsd to listen on localhost
copy:
template:
dest: /etc/nut/upsd.conf
content: |
# Managed by Ansible
LISTEN 127.0.0.1 3493
src: templates/upsd.conf.j2
owner: root
group: nut
mode: "0640"
notify: Restart NUT services
- name: Configure upsd users
copy:
template:
dest: /etc/nut/upsd.users
content: |
# Managed by Ansible
[{{ ups_user }}]
password = {{ ups_password }}
upsmon master
src: templates/upsd.users.j2
owner: root
group: nut
mode: "0640"
notify: Restart NUT services
- name: Configure upsmon
copy:
template:
dest: /etc/nut/upsmon.conf
content: |
# Managed by Ansible
MONITOR {{ ups_name }}@localhost 1 {{ ups_user }} {{ ups_password }} master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POLLFREQ 5
POLLFREQALERT 5
HOSTSYNC 15
DEADTIME 15
POWERDOWNFLAG /etc/killpower
# Notifications
NOTIFYMSG ONLINE "UPS %s on line power"
NOTIFYMSG ONBATT "UPS %s on battery"
NOTIFYMSG LOWBATT "UPS %s battery is low"
NOTIFYMSG FSD "UPS %s: forced shutdown in progress"
NOTIFYMSG COMMOK "Communications with UPS %s established"
NOTIFYMSG COMMBAD "Communications with UPS %s lost"
NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"
NOTIFYMSG REPLBATT "UPS %s battery needs replacing"
# Log all events to syslog
NOTIFYFLAG ONLINE SYSLOG
NOTIFYFLAG ONBATT SYSLOG
NOTIFYFLAG LOWBATT SYSLOG
NOTIFYFLAG FSD SYSLOG
NOTIFYFLAG COMMOK SYSLOG
NOTIFYFLAG COMMBAD SYSLOG
NOTIFYFLAG SHUTDOWN SYSLOG
NOTIFYFLAG REPLBATT SYSLOG
src: templates/upsmon.conf.j2
owner: root
group: nut
mode: "0640"
@ -250,28 +228,33 @@
- nut-monitor
# ═════════════════════════════════════════════════════════════════════════════
# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11.
# ─────────────────────────────────────────────────────────────────────────────
# UPS heartbeat monitoring.
#
# This play WILL FAIL if run as-is, and that is deliberate: uptime_kuma_username
# and uptime_kuma_password were removed from the vault, so the "Validate Uptime
# Kuma configuration" assert stops it before anything is installed or changed.
# The script decides on-mains/on-battery and says so in its exit code, which
# systemd keeps: systemctl is-failed ups-heartbeat.service
#
# It is kept because the CHECK LOGIC is the durable part — what gets measured,
# the thresholds, and the systemd timer plumbing. When something replaces Uptime
# Kuma, only the push transport needs rewriting; the rest still applies.
# Reporting anywhere else is optional. Set `healthcheck_push_url` and the script
# will also GET it with ?status=up|down; leave it empty and the exit code is
# still the whole answer. Today that URL points at Uptime Kuma, which is still
# running on watchtower but is no longer deployed by Ansible - the credentials
# were retired, the service was not. If it is ever replaced, `healthcheck_push_url`
# is the only thing that needs to change here.
#
# What was being monitored: archive/uptime_kuma/MONITORS.md
# ═════════════════════════════════════════════════════════════════════════════
- name: Setup UPS Heartbeat Monitoring with Uptime Kuma
# NOTE: this play has never actually been applied to nodito. /opt/ups-monitoring
# does not exist and there is no ups-heartbeat timer. What is on the box is a
# hand-written /usr/local/bin/ups-heartbeat.sh - mode 0644, not executable, and
# referenced by no unit and no cron entry, so nothing has ever run it. Its push
# token (uLmCPkLLO4) belongs to a monitor that DOES still exist and answer, so
# that monitor has had no heartbeat since January 2026. It is now
# healthcheck_push_urls.ups in the vault, and running this play is what will
# finally start feeding it.
# ─────────────────────────────────────────────────────────────────────────────
- name: Setup UPS Heartbeat Monitoring
hosts: hypervisor
become: true
vars_files:
- ../../infra_vars.yml
- ../../services_config.yml
- ../../infra_secrets.yml
- nodito_vars.yml
- nodito_secrets.yml
vars:
ups_heartbeat_interval_seconds: 60
@ -281,134 +264,10 @@
ups_monitoring_script_path: "{{ ups_monitoring_script_dir }}/ups_heartbeat.sh"
ups_log_file: "{{ ups_monitoring_script_dir }}/ups_heartbeat.log"
ups_systemd_service_name: ups-heartbeat
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
ntfy_topic: "{{ service_settings.ntfy.topic }}"
# Optional. Empty is fine and is not an error - see the banner above.
healthcheck_push_url: "{{ healthcheck_push_urls.ups | default('') }}"
tasks:
- name: Validate Uptime Kuma configuration
assert:
that:
- uptime_kuma_api_url is defined
- uptime_kuma_api_url != ""
- uptime_kuma_username is defined
- uptime_kuma_username != ""
- uptime_kuma_password is defined
- uptime_kuma_password != ""
fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
- name: Get hostname for monitor identification
command: hostname
register: host_name
changed_when: false
- name: Set monitor name and group based on hostname
set_fact:
monitor_name: "ups-{{ host_name.stdout }}"
monitor_friendly_name: "UPS Status: {{ host_name.stdout }}"
uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
- name: Create Uptime Kuma UPS monitor setup script
copy:
dest: /tmp/setup_uptime_kuma_ups_monitor.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
group_name = sys.argv[4]
monitor_name = sys.argv[5]
monitor_description = sys.argv[6]
interval = int(sys.argv[7])
retries = int(sys.argv[8])
ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts"
api = UptimeKumaApi(api_url, timeout=120, wait_events=2.0)
api.login(username, password)
monitors = api.get_monitors()
notifications = api.get_notifications()
ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
notification_id_list = {}
if ntfy_notification:
notification_id_list[ntfy_notification['id']] = True
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
if not group:
api.add_monitor(type='group', name=group_name)
monitors = api.get_monitors()
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
monitor_data = {
'type': 'push',
'name': monitor_name,
'parent': group['id'],
'interval': interval,
'upsideDown': False, # Normal heartbeat mode: receiving pings = healthy
'maxretries': retries,
'description': monitor_description,
'notificationIDList': notification_id_list
}
if existing_monitor:
api.edit_monitor(existing_monitor['id'], **monitor_data)
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
else:
api.add_monitor(**monitor_data)
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
result = {
'monitor_id': monitor['id'],
'push_token': monitor['pushToken'],
'group_name': group_name,
'group_id': group['id'],
'monitor_name': monitor_name
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma UPS monitor setup script
command: >
{{ ansible_playbook_python }}
/tmp/setup_uptime_kuma_ups_monitor.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"{{ uptime_kuma_monitor_group }}"
"{{ monitor_name }}"
"{{ monitor_friendly_name }} - Alerts when UPS goes on battery or loses communication"
"{{ ups_heartbeat_timeout_seconds }}"
"{{ ups_heartbeat_retries }}"
"{{ ntfy_topic }}"
register: monitor_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse monitor setup result
set_fact:
monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
- name: Set push URL as fact
set_fact:
uptime_kuma_ups_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
- name: Install required packages for UPS monitoring
package:
name:
@ -424,107 +283,25 @@
mode: '0755'
- name: Create UPS heartbeat monitoring script
copy:
template:
dest: "{{ ups_monitoring_script_path }}"
content: |
#!/bin/bash
# UPS Heartbeat Monitoring Script
# Sends heartbeat to Uptime Kuma only when UPS is on mains power
# When on battery or communication lost, no heartbeat is sent (triggers timeout alert)
LOG_FILE="{{ ups_log_file }}"
UPTIME_KUMA_URL="{{ uptime_kuma_ups_push_url }}"
UPS_NAME="{{ ups_name }}"
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
send_heartbeat() {
local message="$1"
local encoded_message
encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g; s/%/%25/g')
local response http_code
response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Heartbeat sent: $message (HTTP $http_code)"
return 0
else
log_message "ERROR: Failed to send heartbeat (HTTP $http_code)"
return 1
fi
}
main() {
local status charge runtime load
status=$(upsc ${UPS_NAME}@localhost ups.status 2>/dev/null)
if [ -z "$status" ]; then
log_message "ERROR: Cannot communicate with UPS - NOT sending heartbeat"
exit 1
fi
charge=$(upsc ${UPS_NAME}@localhost battery.charge 2>/dev/null)
runtime=$(upsc ${UPS_NAME}@localhost battery.runtime 2>/dev/null)
load=$(upsc ${UPS_NAME}@localhost ups.load 2>/dev/null)
if [[ "$status" == *"OL"* ]]; then
local message="UPS on mains (charge=${charge}% runtime=${runtime}s load=${load}%)"
send_heartbeat "$message"
exit 0
else
log_message "UPS not on mains power (status=$status) - NOT sending heartbeat"
exit 1
fi
}
main
src: templates/ups_heartbeat.sh.j2
owner: root
group: root
mode: '0755'
- name: Create systemd service for UPS heartbeat
copy:
template:
dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.service"
content: |
[Unit]
Description=UPS Heartbeat Monitor
After=network.target nut-monitor.service
[Service]
Type=oneshot
ExecStart={{ ups_monitoring_script_path }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
src: templates/ups-heartbeat.service.j2
owner: root
group: root
mode: '0644'
- name: Create systemd timer for UPS heartbeat
copy:
template:
dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.timer"
content: |
[Unit]
Description=Run UPS Heartbeat Monitor every {{ ups_heartbeat_interval_seconds }} seconds
Requires={{ ups_systemd_service_name }}.service
[Timer]
OnBootSec=1min
OnUnitActiveSec={{ ups_heartbeat_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target
src: templates/ups-heartbeat.timer.j2
owner: root
group: root
mode: '0644'
@ -561,22 +338,12 @@
- " Off Delay: {{ ups_offdelay }}s (time after shutdown before UPS cuts power)"
- " On Delay: {{ ups_ondelay }}s (time after mains returns before UPS restores power)"
- ""
- "Uptime Kuma Monitoring:"
- " Monitor Name: {{ monitor_friendly_name }}"
- " Monitor Group: {{ uptime_kuma_monitor_group }}"
- " Push URL: {{ uptime_kuma_ups_push_url }}"
- " Heartbeat Interval: {{ ups_heartbeat_interval_seconds }}s"
- " Timeout: {{ ups_heartbeat_timeout_seconds }}s"
- "Health reporting:"
- " Check interval: {{ ups_heartbeat_interval_seconds }}s"
- " Push URL: {{ healthcheck_push_url | default('', true) | ternary('set', 'not set - exit code only') }}"
- ""
- "Scripts and Services:"
- " Script: {{ ups_monitoring_script_path }}"
- " Log: {{ ups_log_file }}"
- " Service: {{ ups_systemd_service_name }}.service"
- " Timer: {{ ups_systemd_service_name }}.timer"
- name: Clean up temporary Uptime Kuma setup script
file:
path: /tmp/setup_uptime_kuma_ups_monitor.py
state: absent
delegate_to: localhost
become: no

12
ansible/infra/nodito/nodito_secrets.yml
View file

@ -1,12 +0,0 @@
$ANSIBLE_VAULT;1.1;AES256
39386134336137333139343861353965353665626161303634323563386563356364373265616163
3237623438636161323366313363356335613566303338630a663732316663333136653737663935
34373466316538663962636134303530323837316563336136653636643430363566376363323666
3531613636393331310a306633613030343138656237663061653438336562323135633732356563
33396136656564353465636361613161326465303166373131343562363338303966666362643532
33383561366539393930633766363533313363653733393263376634316362643863376635366638
36396333633231616662323465323932656565396138313264613832346261616231333265636439
61666334613662373839613833613663333436373365376534643662656335316536303739616437
65333635613330353536353162646234323266316338643435653864666364313734386665303830
34383961633734646134373866623330663038366130306265656466653562643764346162313333
353664363335303433393330306332666336

28
ansible/infra/nodito/nodito_vars.yml
View file

@ -1,28 +0,0 @@
# Nodito CPU Temperature Monitoring Configuration
# Temperature Monitoring Configuration
temp_threshold_celsius: 80
temp_check_interval_minutes: 1
# Script Configuration
monitoring_script_dir: /opt/nodito-monitoring
monitoring_script_path: "{{ monitoring_script_dir }}/cpu_temp_monitor.sh"
log_file: "{{ monitoring_script_dir }}/cpu_temp_monitor.log"
# System Configuration
systemd_service_name: nodito-cpu-temp-monitor
# ZFS Pool Configuration
zfs_pool_name: "proxmox-tank-1"
zfs_disk_1: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN0Z" # First disk for RAID 1 mirror
zfs_disk_2: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P" # Second disk for RAID 1 mirror
zfs_pool_mountpoint: "/var/lib/vz"
# UPS Configuration (CyberPower CP900EPFCLCD via USB)
ups_name: cyberpower
ups_desc: "CyberPower CP900EPFCLCD"
ups_driver: usbhid-ups
ups_port: auto
ups_user: counterweight
ups_offdelay: 120 # Seconds after shutdown before UPS cuts outlet power
ups_ondelay: 30 # Seconds after mains returns before UPS restores outlet power

2
ansible/infra/nodito/templates/nut.conf.j2 Normal file
View file

@ -0,0 +1,2 @@
# Managed by Ansible
MODE=standalone

14
ansible/infra/nodito/templates/ups-heartbeat.service.j2 Normal file
View file

@ -0,0 +1,14 @@
[Unit]
Description=UPS Heartbeat Monitor
After=network.target nut-monitor.service
[Service]
Type=oneshot
ExecStart={{ ups_monitoring_script_path }}
User=root
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

11
ansible/infra/nodito/templates/ups-heartbeat.timer.j2 Normal file
View file

@ -0,0 +1,11 @@
[Unit]
Description=Run UPS Heartbeat Monitor every {{ ups_heartbeat_interval_seconds }} seconds
Requires={{ ups_systemd_service_name }}.service
[Timer]
OnBootSec=1min
OnUnitActiveSec={{ ups_heartbeat_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target

9
ansible/infra/nodito/templates/ups.conf.j2 Normal file
View file

@ -0,0 +1,9 @@
# Managed by Ansible
maxretry = 3
[{{ ups_name }}]
driver = {{ ups_driver }}
port = {{ ups_port }}
desc = "{{ ups_desc }}"
offdelay = {{ ups_offdelay }}
ondelay = {{ ups_ondelay }}

68
ansible/infra/nodito/templates/ups_heartbeat.sh.j2 Normal file
View file

@ -0,0 +1,68 @@
#!/bin/bash
# UPS heartbeat check - managed by Ansible (infra/nodito/34_nut_ups_setup_playbook.yml)
#
# The exit code is the answer and systemd keeps it:
# systemctl is-failed {{ ups_systemd_service_name }}.service
# Reporting anywhere else is optional and generic.
LOG_FILE="{{ ups_log_file }}"
UPS_NAME="{{ ups_name }}"
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
report() {
local status="$1"
local message="$2"
# No push URL is normal, not an error: the exit code below is still a
# complete answer for anything reading unit state.
[ -n "$PUSH_URL" ] || return 0
local encoded_message
encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
local response http_code
response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Reported ${status}: $message (HTTP $http_code)"
return 0
else
log_message "ERROR: Failed to report ${status} (HTTP $http_code)"
return 1
fi
}
main() {
local status charge runtime load
status=$(upsc ${UPS_NAME}@localhost ups.status 2>/dev/null)
if [ -z "$status" ]; then
log_message "ERROR: Cannot communicate with UPS"
report "down" "cannot communicate with UPS ${UPS_NAME}"
exit 1
fi
charge=$(upsc ${UPS_NAME}@localhost battery.charge 2>/dev/null)
runtime=$(upsc ${UPS_NAME}@localhost battery.runtime 2>/dev/null)
load=$(upsc ${UPS_NAME}@localhost ups.load 2>/dev/null)
if [[ "$status" == *"OL"* ]]; then
local message="UPS on mains (charge=${charge}% runtime=${runtime}s load=${load}%)"
log_message "$message"
report "up" "$message"
exit 0
else
log_message "UPS not on mains power (status=$status)"
report "down" "UPS not on mains (status=${status} charge=${charge}%)"
exit 1
fi
}
main

2
ansible/infra/nodito/templates/upsd.conf.j2 Normal file
View file

@ -0,0 +1,2 @@
# Managed by Ansible
LISTEN 127.0.0.1 3493

4
ansible/infra/nodito/templates/upsd.users.j2 Normal file
View file

@ -0,0 +1,4 @@
# Managed by Ansible
[{{ ups_user }}]
password = {{ ups_password }}
upsmon master

34
ansible/infra/nodito/templates/upsmon.conf.j2 Normal file
View file

@ -0,0 +1,34 @@
# Managed by Ansible
MONITOR {{ ups_name }}@localhost 1 {{ ups_user }} {{ ups_password }} master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POLLFREQ 5
POLLFREQALERT 5
HOSTSYNC 15
DEADTIME 15
POWERDOWNFLAG "/etc/killpower"
OFFDURATION 30
RBWARNTIME 43200
NOCOMMWARNTIME 300
FINALDELAY 5
# Notifications
NOTIFYMSG ONLINE "UPS %s on line power"
NOTIFYMSG ONBATT "UPS %s on battery"
NOTIFYMSG LOWBATT "UPS %s battery is low"
NOTIFYMSG FSD "UPS %s: forced shutdown in progress"
NOTIFYMSG COMMOK "Communications with UPS %s established"
NOTIFYMSG COMMBAD "Communications with UPS %s lost"
NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"
NOTIFYMSG REPLBATT "UPS %s battery needs replacing"
# Log all events to syslog
NOTIFYFLAG ONLINE SYSLOG
NOTIFYFLAG ONBATT SYSLOG
NOTIFYFLAG LOWBATT SYSLOG
NOTIFYFLAG FSD SYSLOG
NOTIFYFLAG COMMOK SYSLOG
NOTIFYFLAG COMMBAD SYSLOG
NOTIFYFLAG SHUTDOWN SYSLOG
NOTIFYFLAG REPLBATT SYSLOG

14
ansible/infra/nodito/templates/zfs-health-monitor.service.j2 Normal file
View file

@ -0,0 +1,14 @@
[Unit]
Description=ZFS Pool Health Monitor
After=zfs.target network.target
[Service]
Type=oneshot
ExecStart={{ zfs_monitoring_script_path }}
User=root
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

11
ansible/infra/nodito/templates/zfs-health-monitor.timer.j2 Normal file
View file

@ -0,0 +1,11 @@
[Unit]
Description=Run ZFS Pool Health Monitor daily
Requires={{ zfs_systemd_health_service_name }}.service
[Timer]
OnBootSec=5min
OnUnitActiveSec={{ zfs_check_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target

13
ansible/infra/nodito/templates/zfs-monthly-scrub.service.j2 Normal file
View file

@ -0,0 +1,13 @@
[Unit]
Description=ZFS Monthly Scrub for {{ zfs_pool_name }}
After=zfs.target
[Service]
Type=oneshot
ExecStart=/sbin/zpool scrub {{ zfs_pool_name }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

10
ansible/infra/nodito/templates/zfs-monthly-scrub.timer.j2 Normal file
View file

@ -0,0 +1,10 @@
[Unit]
Description=Run ZFS Scrub on last day of every month at 4:00 AM
Requires={{ zfs_systemd_scrub_service_name }}.service
[Timer]
OnCalendar=*-*~01 04:00:00
Persistent=true
[Install]
WantedBy=timers.target

181
ansible/infra/nodito/templates/zfs_health_monitor.sh.j2 Normal file
View file

@ -0,0 +1,181 @@
#!/bin/bash
# ZFS pool health check - managed by Ansible (infra/nodito/32_zfs_pool_setup_playbook.yml)
#
# The exit code is the answer and systemd keeps it:
# systemctl is-failed {{ zfs_systemd_health_service_name }}.service
# Reporting anywhere else is optional and generic.
LOG_FILE="{{ zfs_log_file }}"
POOL_NAME="{{ zfs_pool_name }}"
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
HOSTNAME=$(hostname)
# Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
# Function to check pool health using JSON output
check_pool_health() {
local pool="$1"
local issues_found=0
# Get pool status as JSON
local pool_json
pool_json=$(zpool status -j "$pool" 2>&1)
if [ $? -ne 0 ]; then
log_message "ERROR: Failed to get pool status for $pool"
log_message " -> $pool_json"
return 1
fi
# Check 1: Pool state must be ONLINE
local pool_state
pool_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].state')
if [ "$pool_state" != "ONLINE" ]; then
log_message "ISSUE: Pool state is $pool_state (expected ONLINE)"
issues_found=1
else
log_message "OK: Pool state is ONLINE"
fi
# Check 2: Check all vdevs and devices for non-ONLINE states
local bad_states
bad_states=$(echo "$pool_json" | jq -r --arg pool "$pool" '
.pools[$pool].vdevs[] |
.. | objects |
select(.state? and .state != "ONLINE") |
"\(.name // "unknown"): \(.state)"
' 2>/dev/null)
if [ -n "$bad_states" ]; then
log_message "ISSUE: Found devices not in ONLINE state:"
echo "$bad_states" | while read -r line; do
log_message " -> $line"
done
issues_found=1
else
log_message "OK: All devices are ONLINE"
fi
# Check 3: Check for resilvering in progress
local scan_function scan_state
scan_function=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
if [ "$scan_function" = "RESILVER" ] && [ "$scan_state" = "SCANNING" ]; then
local resilver_progress
resilver_progress=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.issued // "unknown"')
log_message "ISSUE: Pool is currently resilvering (disk reconstruction in progress) - ${resilver_progress} processed"
issues_found=1
fi
# Check 4: Check for read/write/checksum errors on all devices
# Note: ZFS JSON output has error counts as strings, so convert to numbers for comparison
local devices_with_errors
devices_with_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '
.pools[$pool].vdevs[] |
.. | objects |
select(.name? and ((.read_errors // "0" | tonumber) > 0 or (.write_errors // "0" | tonumber) > 0 or (.checksum_errors // "0" | tonumber) > 0)) |
"\(.name): read=\(.read_errors // 0) write=\(.write_errors // 0) cksum=\(.checksum_errors // 0)"
' 2>/dev/null)
if [ -n "$devices_with_errors" ]; then
log_message "ISSUE: Found devices with I/O errors:"
echo "$devices_with_errors" | while read -r line; do
log_message " -> $line"
done
issues_found=1
else
log_message "OK: No read/write/checksum errors detected"
fi
# Check 5: Check for scan errors (from last scrub/resilver)
local scan_errors
scan_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.errors // "0"')
if [ "$scan_errors" != "0" ] && [ "$scan_errors" != "null" ] && [ -n "$scan_errors" ]; then
log_message "ISSUE: Last scan reported $scan_errors errors"
issues_found=1
else
log_message "OK: No scan errors"
fi
return $issues_found
}
# Function to get last scrub info for status message
get_scrub_info() {
local pool="$1"
local pool_json
pool_json=$(zpool status -j "$pool" 2>/dev/null)
local scan_func scan_state scan_start
scan_func=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
scan_start=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.start_time // ""')
if [ "$scan_func" = "SCRUB" ] && [ "$scan_state" = "SCANNING" ]; then
echo "scrub in progress (started $scan_start)"
elif [ "$scan_func" = "SCRUB" ] && [ -n "$scan_start" ]; then
echo "last scrub: $scan_start"
else
echo "no scrub history"
fi
}
# Optional reporting to whatever is watching. No push URL is normal, not an
# error: the script's exit code is still a complete answer for anything reading
# unit state.
report() {
local status="$1"
local message="$2"
[ -n "$PUSH_URL" ] || return 0
log_message "Reporting ${status}: $message"
# URL encode the message
local encoded_message
encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
local response http_code
response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Report sent successfully (HTTP $http_code)"
return 0
else
log_message "ERROR: Failed to report (HTTP $http_code)"
return 1
fi
}
# Main health check logic
main() {
log_message "=========================================="
log_message "Starting ZFS health check for pool: $POOL_NAME on $HOSTNAME"
# Run all health checks
if check_pool_health "$POOL_NAME"; then
local scrub_info
scrub_info=$(get_scrub_info "$POOL_NAME")
local message="Pool $POOL_NAME healthy ($scrub_info)"
report "up" "$message"
log_message "Health check completed: ALL OK"
exit 0
else
log_message "Health check completed: ISSUES DETECTED"
report "down" "Pool $POOL_NAME unhealthy - see $LOG_FILE"
exit 1
fi
}
# Run main function
main