nodito: de-Uptime-Kuma the ZFS and NUT playbooks, extract templates

These two are host-specific by design - nodito is a pet, not cattle - so they
stay playbooks rather than becoming roles. But they had rotted.

De-Kuma, following the pattern of the six service roles:

  Both plays opened with an assert on uptime_kuma_username/password, which were
  removed from the vault, so both failed before doing anything. Dropped that,
  the two embedded Python monitor-creation scripts, and their /tmp cleanup.
  Kept every check, threshold and systemd timer - those are the durable part.

  Reporting is now generic: `healthcheck_push_url` goes into the unit as
  Environment=HEALTHCHECK_PUSH_URL and the script reads ${HEALTHCHECK_PUSH_URL:-},
  treating empty as normal rather than an error. The exit code is the real
  answer; systemd keeps it. Both scripts now also report status=down on failure
  instead of only going silent. Live push URLs harvested into the vault so
  nothing observable changes for ZFS.

Three live bugs found while check-diffing:

  1. 32_zfs would have DE-REGISTERED the Proxmox storage. `pvesm remove` was
     gated on the storage existing and `pvesm add` on it NOT existing - mutually
     exclusive - so a real run removed the proxmox-tank-1 entry backing every VM
     and never put it back. It would also have dropped `mountpoint /var/lib/vz`,
     which the live entry has and `pvesm add` does not set. Registration is now
     add-only.

  2. zfs_disk_1 named ata-...WX11TN0Z, a disk no longer in the machine. The live
     mirror is WX120LHQ + WX11TN2P; a leg was replaced and the repo never caught
     up. Inert behind `when: zfs_pool_exists.rc != 0`, but wrong on any
     disaster-recovery run. This is the seventh instance of an identifier
     written down once whose hardware later moved.

  3. 34_nut has NEVER been applied to nodito - no /etc/nut file carries the
     "Managed by Ansible" marker; they were written by hand in January 2026. The
     vault held the literal CHANGE_ME_TO_SECURE_PASSWORD, so applying it would
     have overwritten a working upsd/upsmon auth pair with a placeholder and
     restarted NUT, leaving the hypervisor's UPS unable to trigger a clean
     shutdown on mains loss. The Kuma assert was the only thing stopping that,
     so removing it without a replacement would have armed the gun: there is now
     an explicit assert that refuses to run on the placeholder. The real
     password is in the vault and `Configure upsd users` check-diffs clean.

  Templates reconciled with the live files first, so applying 34_nut is close to
  a no-op: added `maxretry = 3` to ups.conf and OFFDURATION / RBWARNTIME /
  NOCOMMWARNTIME / FINALDELAY plus quoted POWERDOWNFLAG to upsmon.conf. The only
  substantive additions left are the NOTIFYMSG/NOTIFYFLAG syslog lines.

  /usr/local/bin/ups-heartbeat.sh on the box is an orphan - mode 0644, not
  executable, referenced by no unit and no cron entry - but its push token
  belongs to a monitor that still exists and answers, so that monitor has had no
  heartbeat since January. Harvested as healthcheck_push_urls.ups; applying this
  play is what will finally feed it.

Finish the host_vars migration:

  infra/nodito/nodito_vars.yml was byte-identical to host_vars/nodito/main.yml.
  Deleted it, moved nodito_secrets.yml to host_vars/nodito/vault.yml, and
  stripped the dead vars_files entries from all three playbooks.

Extract the 13 inline `content: |` blocks to infra/nodito/templates/, pulled via
a YAML load rather than retyped. 681+582 lines become 313+344 plus templates.
Ownership parity against HEAD checked mechanically: no owner/group/mode drift on
any surviving task.

Neither playbook has been applied. ZFS play 2 check-runs failed=0 with two
changes: the script rewrite (the deployed one is a hand-edited DEBUG VERSION
with `set -x`) and the Environment line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/infra/nodito/templates/nut.conf.j2

@@ -0,0 +1,2 @@
+# Managed by Ansible
+MODE=standalone

ansible/infra/nodito/templates/ups-heartbeat.service.j2

@@ -0,0 +1,14 @@
+[Unit]
+Description=UPS Heartbeat Monitor
+After=network.target nut-monitor.service
+
+[Service]
+Type=oneshot
+ExecStart={{ ups_monitoring_script_path }}
+User=root
+Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

ansible/infra/nodito/templates/ups-heartbeat.timer.j2

@@ -0,0 +1,11 @@
+[Unit]
+Description=Run UPS Heartbeat Monitor every {{ ups_heartbeat_interval_seconds }} seconds
+Requires={{ ups_systemd_service_name }}.service
+
+[Timer]
+OnBootSec=1min
+OnUnitActiveSec={{ ups_heartbeat_interval_seconds }}sec
+Persistent=true
+
+[Install]
+WantedBy=timers.target

ansible/infra/nodito/templates/ups.conf.j2

@@ -0,0 +1,9 @@
+# Managed by Ansible
+maxretry = 3
+
+[{{ ups_name }}]
+    driver = {{ ups_driver }}
+    port = {{ ups_port }}
+    desc = "{{ ups_desc }}"
+    offdelay = {{ ups_offdelay }}
+    ondelay = {{ ups_ondelay }}

ansible/infra/nodito/templates/ups_heartbeat.sh.j2

@@ -0,0 +1,68 @@
+#!/bin/bash
+
+# UPS heartbeat check - managed by Ansible (infra/nodito/34_nut_ups_setup_playbook.yml)
+#
+# The exit code is the answer and systemd keeps it:
+#     systemctl is-failed {{ ups_systemd_service_name }}.service
+# Reporting anywhere else is optional and generic.
+
+LOG_FILE="{{ ups_log_file }}"
+UPS_NAME="{{ ups_name }}"
+PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
+
+log_message() {
+    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
+}
+
+report() {
+    local status="$1"
+    local message="$2"
+
+    # No push URL is normal, not an error: the exit code below is still a
+    # complete answer for anything reading unit state.
+    [ -n "$PUSH_URL" ] || return 0
+
+    local encoded_message
+    encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
+
+    local response http_code
+    response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
+    http_code=$(echo "$response" | tail -n1)
+
+    if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
+        log_message "Reported ${status}: $message (HTTP $http_code)"
+        return 0
+    else
+        log_message "ERROR: Failed to report ${status} (HTTP $http_code)"
+        return 1
+    fi
+}
+
+main() {
+    local status charge runtime load
+
+    status=$(upsc ${UPS_NAME}@localhost ups.status 2>/dev/null)
+
+    if [ -z "$status" ]; then
+        log_message "ERROR: Cannot communicate with UPS"
+        report "down" "cannot communicate with UPS ${UPS_NAME}"
+        exit 1
+    fi
+
+    charge=$(upsc ${UPS_NAME}@localhost battery.charge 2>/dev/null)
+    runtime=$(upsc ${UPS_NAME}@localhost battery.runtime 2>/dev/null)
+    load=$(upsc ${UPS_NAME}@localhost ups.load 2>/dev/null)
+
+    if [[ "$status" == *"OL"* ]]; then
+        local message="UPS on mains (charge=${charge}% runtime=${runtime}s load=${load}%)"
+        log_message "$message"
+        report "up" "$message"
+        exit 0
+    else
+        log_message "UPS not on mains power (status=$status)"
+        report "down" "UPS not on mains (status=${status} charge=${charge}%)"
+        exit 1
+    fi
+}
+
+main

ansible/infra/nodito/templates/upsd.conf.j2

@@ -0,0 +1,2 @@
+# Managed by Ansible
+LISTEN 127.0.0.1 3493

ansible/infra/nodito/templates/upsd.users.j2

@@ -0,0 +1,4 @@
+# Managed by Ansible
+[{{ ups_user }}]
+    password = {{ ups_password }}
+    upsmon master

ansible/infra/nodito/templates/upsmon.conf.j2

@@ -0,0 +1,34 @@
+# Managed by Ansible
+MONITOR {{ ups_name }}@localhost 1 {{ ups_user }} {{ ups_password }} master
+
+MINSUPPLIES 1
+SHUTDOWNCMD "/sbin/shutdown -h +0"
+POLLFREQ 5
+POLLFREQALERT 5
+HOSTSYNC 15
+DEADTIME 15
+POWERDOWNFLAG "/etc/killpower"
+OFFDURATION 30
+RBWARNTIME 43200
+NOCOMMWARNTIME 300
+FINALDELAY 5
+
+# Notifications
+NOTIFYMSG ONLINE    "UPS %s on line power"
+NOTIFYMSG ONBATT    "UPS %s on battery"
+NOTIFYMSG LOWBATT   "UPS %s battery is low"
+NOTIFYMSG FSD       "UPS %s: forced shutdown in progress"
+NOTIFYMSG COMMOK    "Communications with UPS %s established"
+NOTIFYMSG COMMBAD   "Communications with UPS %s lost"
+NOTIFYMSG SHUTDOWN  "Auto logout and shutdown proceeding"
+NOTIFYMSG REPLBATT  "UPS %s battery needs replacing"
+
+# Log all events to syslog
+NOTIFYFLAG ONLINE   SYSLOG
+NOTIFYFLAG ONBATT   SYSLOG
+NOTIFYFLAG LOWBATT  SYSLOG
+NOTIFYFLAG FSD      SYSLOG
+NOTIFYFLAG COMMOK   SYSLOG
+NOTIFYFLAG COMMBAD  SYSLOG
+NOTIFYFLAG SHUTDOWN SYSLOG
+NOTIFYFLAG REPLBATT SYSLOG

ansible/infra/nodito/templates/zfs-health-monitor.service.j2

@@ -0,0 +1,14 @@
+[Unit]
+Description=ZFS Pool Health Monitor
+After=zfs.target network.target
+
+[Service]
+Type=oneshot
+ExecStart={{ zfs_monitoring_script_path }}
+User=root
+Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

ansible/infra/nodito/templates/zfs-health-monitor.timer.j2

@@ -0,0 +1,11 @@
+[Unit]
+Description=Run ZFS Pool Health Monitor daily
+Requires={{ zfs_systemd_health_service_name }}.service
+
+[Timer]
+OnBootSec=5min
+OnUnitActiveSec={{ zfs_check_interval_seconds }}sec
+Persistent=true
+
+[Install]
+WantedBy=timers.target

ansible/infra/nodito/templates/zfs-monthly-scrub.service.j2

@@ -0,0 +1,13 @@
+[Unit]
+Description=ZFS Monthly Scrub for {{ zfs_pool_name }}
+After=zfs.target
+
+[Service]
+Type=oneshot
+ExecStart=/sbin/zpool scrub {{ zfs_pool_name }}
+User=root
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

ansible/infra/nodito/templates/zfs-monthly-scrub.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Run ZFS Scrub on last day of every month at 4:00 AM
+Requires={{ zfs_systemd_scrub_service_name }}.service
+
+[Timer]
+OnCalendar=*-*~01 04:00:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target

ansible/infra/nodito/templates/zfs_health_monitor.sh.j2

@@ -0,0 +1,181 @@
+#!/bin/bash
+
+# ZFS pool health check - managed by Ansible (infra/nodito/32_zfs_pool_setup_playbook.yml)
+#
+# The exit code is the answer and systemd keeps it:
+#     systemctl is-failed {{ zfs_systemd_health_service_name }}.service
+# Reporting anywhere else is optional and generic.
+
+LOG_FILE="{{ zfs_log_file }}"
+POOL_NAME="{{ zfs_pool_name }}"
+PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
+HOSTNAME=$(hostname)
+
+# Function to log messages
+log_message() {
+    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
+}
+
+# Function to check pool health using JSON output
+check_pool_health() {
+    local pool="$1"
+    local issues_found=0
+    
+    # Get pool status as JSON
+    local pool_json
+    pool_json=$(zpool status -j "$pool" 2>&1)
+    
+    if [ $? -ne 0 ]; then
+        log_message "ERROR: Failed to get pool status for $pool"
+        log_message "  -> $pool_json"
+        return 1
+    fi
+    
+    # Check 1: Pool state must be ONLINE
+    local pool_state
+    pool_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].state')
+    
+    if [ "$pool_state" != "ONLINE" ]; then
+        log_message "ISSUE: Pool state is $pool_state (expected ONLINE)"
+        issues_found=1
+    else
+        log_message "OK: Pool state is ONLINE"
+    fi
+    
+    # Check 2: Check all vdevs and devices for non-ONLINE states
+    local bad_states
+    bad_states=$(echo "$pool_json" | jq -r --arg pool "$pool" '
+        .pools[$pool].vdevs[] | 
+        .. | objects | 
+        select(.state? and .state != "ONLINE") | 
+        "\(.name // "unknown"): \(.state)"
+    ' 2>/dev/null)
+    
+    if [ -n "$bad_states" ]; then
+        log_message "ISSUE: Found devices not in ONLINE state:"
+        echo "$bad_states" | while read -r line; do
+            log_message "  -> $line"
+        done
+        issues_found=1
+    else
+        log_message "OK: All devices are ONLINE"
+    fi
+    
+    # Check 3: Check for resilvering in progress
+    local scan_function scan_state
+    scan_function=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
+    scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
+    
+    if [ "$scan_function" = "RESILVER" ] && [ "$scan_state" = "SCANNING" ]; then
+        local resilver_progress
+        resilver_progress=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.issued // "unknown"')
+        log_message "ISSUE: Pool is currently resilvering (disk reconstruction in progress) - ${resilver_progress} processed"
+        issues_found=1
+    fi
+    
+    # Check 4: Check for read/write/checksum errors on all devices
+    # Note: ZFS JSON output has error counts as strings, so convert to numbers for comparison
+    local devices_with_errors
+    devices_with_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '
+        .pools[$pool].vdevs[] | 
+        .. | objects | 
+        select(.name? and ((.read_errors // "0" | tonumber) > 0 or (.write_errors // "0" | tonumber) > 0 or (.checksum_errors // "0" | tonumber) > 0)) | 
+        "\(.name): read=\(.read_errors // 0) write=\(.write_errors // 0) cksum=\(.checksum_errors // 0)"
+    ' 2>/dev/null)
+    
+    if [ -n "$devices_with_errors" ]; then
+        log_message "ISSUE: Found devices with I/O errors:"
+        echo "$devices_with_errors" | while read -r line; do
+            log_message "  -> $line"
+        done
+        issues_found=1
+    else
+        log_message "OK: No read/write/checksum errors detected"
+    fi
+    
+    # Check 5: Check for scan errors (from last scrub/resilver)
+    local scan_errors
+    scan_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.errors // "0"')
+    
+    if [ "$scan_errors" != "0" ] && [ "$scan_errors" != "null" ] && [ -n "$scan_errors" ]; then
+        log_message "ISSUE: Last scan reported $scan_errors errors"
+        issues_found=1
+    else
+        log_message "OK: No scan errors"
+    fi
+    
+    return $issues_found
+}
+
+# Function to get last scrub info for status message
+get_scrub_info() {
+    local pool="$1"
+    local pool_json
+    pool_json=$(zpool status -j "$pool" 2>/dev/null)
+    
+    local scan_func scan_state scan_start
+    scan_func=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
+    scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
+    scan_start=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.start_time // ""')
+    
+    if [ "$scan_func" = "SCRUB" ] && [ "$scan_state" = "SCANNING" ]; then
+        echo "scrub in progress (started $scan_start)"
+    elif [ "$scan_func" = "SCRUB" ] && [ -n "$scan_start" ]; then
+        echo "last scrub: $scan_start"
+    else
+        echo "no scrub history"
+    fi
+}
+
+# Optional reporting to whatever is watching. No push URL is normal, not an
+# error: the script's exit code is still a complete answer for anything reading
+# unit state.
+report() {
+    local status="$1"
+    local message="$2"
+    
+    [ -n "$PUSH_URL" ] || return 0
+    
+    log_message "Reporting ${status}: $message"
+    
+    # URL encode the message
+    local encoded_message
+    encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
+    
+    local response http_code
+    response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
+    http_code=$(echo "$response" | tail -n1)
+    
+    if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
+        log_message "Report sent successfully (HTTP $http_code)"
+        return 0
+    else
+        log_message "ERROR: Failed to report (HTTP $http_code)"
+        return 1
+    fi
+}
+
+# Main health check logic
+main() {
+    log_message "=========================================="
+    log_message "Starting ZFS health check for pool: $POOL_NAME on $HOSTNAME"
+    
+    # Run all health checks
+    if check_pool_health "$POOL_NAME"; then
+        local scrub_info
+        scrub_info=$(get_scrub_info "$POOL_NAME")
+        
+        local message="Pool $POOL_NAME healthy ($scrub_info)"
+        report "up" "$message"
+        
+        log_message "Health check completed: ALL OK"
+        exit 0
+    else
+        log_message "Health check completed: ISSUES DETECTED"
+        report "down" "Pool $POOL_NAME unhealthy - see $LOG_FILE"
+        exit 1
+    fi
+}
+
+# Run main function
+main