counterweight
0f03c503c8
These two are host-specific by design - nodito is a pet, not cattle - so they
stay playbooks rather than becoming roles. But they had rotted.
De-Kuma, following the pattern of the six service roles:
Both plays opened with an assert on uptime_kuma_username/password, which were
removed from the vault, so both failed before doing anything. Dropped that,
the two embedded Python monitor-creation scripts, and their /tmp cleanup.
Kept every check, threshold and systemd timer - those are the durable part.
Reporting is now generic: `healthcheck_push_url` goes into the unit as
Environment=HEALTHCHECK_PUSH_URL and the script reads ${HEALTHCHECK_PUSH_URL:-},
treating empty as normal rather than an error. The exit code is the real
answer; systemd keeps it. Both scripts now also report status=down on failure
instead of only going silent. Live push URLs harvested into the vault so
nothing observable changes for ZFS.
Three live bugs found while check-diffing:
1. 32_zfs would have DE-REGISTERED the Proxmox storage. `pvesm remove` was
gated on the storage existing and `pvesm add` on it NOT existing - mutually
exclusive - so a real run removed the proxmox-tank-1 entry backing every VM
and never put it back. It would also have dropped `mountpoint /var/lib/vz`,
which the live entry has and `pvesm add` does not set. Registration is now
add-only.
2. zfs_disk_1 named ata-...WX11TN0Z, a disk no longer in the machine. The live
mirror is WX120LHQ + WX11TN2P; a leg was replaced and the repo never caught
up. Inert behind `when: zfs_pool_exists.rc != 0`, but wrong on any
disaster-recovery run. This is the seventh instance of an identifier
written down once whose hardware later moved.
3. 34_nut has NEVER been applied to nodito - no /etc/nut file carries the
"Managed by Ansible" marker; they were written by hand in January 2026. The
vault held the literal CHANGE_ME_TO_SECURE_PASSWORD, so applying it would
have overwritten a working upsd/upsmon auth pair with a placeholder and
restarted NUT, leaving the hypervisor's UPS unable to trigger a clean
shutdown on mains loss. The Kuma assert was the only thing stopping that,
so removing it without a replacement would have armed the gun: there is now
an explicit assert that refuses to run on the placeholder. The real
password is in the vault and `Configure upsd users` check-diffs clean.
Templates reconciled with the live files first, so applying 34_nut is close to
a no-op: added `maxretry = 3` to ups.conf and OFFDURATION / RBWARNTIME /
NOCOMMWARNTIME / FINALDELAY plus quoted POWERDOWNFLAG to upsmon.conf. The only
substantive additions left are the NOTIFYMSG/NOTIFYFLAG syslog lines.
/usr/local/bin/ups-heartbeat.sh on the box is an orphan - mode 0644, not
executable, referenced by no unit and no cron entry - but its push token
belongs to a monitor that still exists and answers, so that monitor has had no
heartbeat since January. Harvested as healthcheck_push_urls.ups; applying this
play is what will finally feed it.
Finish the host_vars migration:
infra/nodito/nodito_vars.yml was byte-identical to host_vars/nodito/main.yml.
Deleted it, moved nodito_secrets.yml to host_vars/nodito/vault.yml, and
stripped the dead vars_files entries from all three playbooks.
Extract the 13 inline `content: |` blocks to infra/nodito/templates/, pulled via
a YAML load rather than retyped. 681+582 lines become 313+344 plus templates.
Ownership parity against HEAD checked mechanically: no owner/group/mode drift on
any surviving task.
Neither playbook has been applied. ZFS play 2 check-runs failed=0 with two
changes: the script rewrite (the deployed one is a hand-edited DEBUG VERSION
with `set -x`) and the Environment line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
68 lines
2.1 KiB
Django/Jinja
68 lines
2.1 KiB
Django/Jinja
#!/bin/bash
|
|
|
|
# UPS heartbeat check - managed by Ansible (infra/nodito/34_nut_ups_setup_playbook.yml)
|
|
#
|
|
# The exit code is the answer and systemd keeps it:
|
|
# systemctl is-failed {{ ups_systemd_service_name }}.service
|
|
# Reporting anywhere else is optional and generic.
|
|
|
|
LOG_FILE="{{ ups_log_file }}"
|
|
UPS_NAME="{{ ups_name }}"
|
|
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
|
|
|
|
log_message() {
|
|
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
|
|
}
|
|
|
|
report() {
|
|
local status="$1"
|
|
local message="$2"
|
|
|
|
# No push URL is normal, not an error: the exit code below is still a
|
|
# complete answer for anything reading unit state.
|
|
[ -n "$PUSH_URL" ] || return 0
|
|
|
|
local encoded_message
|
|
encoded_message=$(printf '%s\n' "$message" | sed 's/%/%25/g; s/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
|
|
|
|
local response http_code
|
|
response=$(curl -s --max-time 10 --retry 2 -w "\n%{http_code}" "${PUSH_URL}?status=${status}&msg=${encoded_message}&ping=" 2>&1)
|
|
http_code=$(echo "$response" | tail -n1)
|
|
|
|
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
|
|
log_message "Reported ${status}: $message (HTTP $http_code)"
|
|
return 0
|
|
else
|
|
log_message "ERROR: Failed to report ${status} (HTTP $http_code)"
|
|
return 1
|
|
fi
|
|
}
|
|
|
|
main() {
|
|
local status charge runtime load
|
|
|
|
status=$(upsc ${UPS_NAME}@localhost ups.status 2>/dev/null)
|
|
|
|
if [ -z "$status" ]; then
|
|
log_message "ERROR: Cannot communicate with UPS"
|
|
report "down" "cannot communicate with UPS ${UPS_NAME}"
|
|
exit 1
|
|
fi
|
|
|
|
charge=$(upsc ${UPS_NAME}@localhost battery.charge 2>/dev/null)
|
|
runtime=$(upsc ${UPS_NAME}@localhost battery.runtime 2>/dev/null)
|
|
load=$(upsc ${UPS_NAME}@localhost ups.load 2>/dev/null)
|
|
|
|
if [[ "$status" == *"OL"* ]]; then
|
|
local message="UPS on mains (charge=${charge}% runtime=${runtime}s load=${load}%)"
|
|
log_message "$message"
|
|
report "up" "$message"
|
|
exit 0
|
|
else
|
|
log_message "UPS not on mains power (status=$status)"
|
|
report "down" "UPS not on mains (status=${status} charge=${charge}%)"
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
main
|