Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
235 lines
7.8 KiB
YAML
235 lines
7.8 KiB
YAML
- name: Setup NUT (Network UPS Tools) for CyberPower UPS
|
|
hosts: hypervisor
|
|
become: true
|
|
|
|
tasks:
|
|
# ------------------------------------------------------------------
|
|
# Safety catch
|
|
#
|
|
# /etc/nut/upsd.users and /etc/nut/upsmon.conf on nodito were written by
|
|
# hand in January 2026 and carry a working password. host_vars/nodito/vault.yml
|
|
# (formerly infra/nodito/nodito_secrets.yml) still holds the literal string
|
|
# CHANGE_ME_TO_SECURE_PASSWORD, so running this play would overwrite that
|
|
# working pair with a placeholder and restart NUT - leaving the hypervisor's
|
|
# UPS unmonitored and unable to trigger a clean shutdown on mains loss.
|
|
#
|
|
# Until the real password is put in the vault, stop here.
|
|
# ansible-vault edit host_vars/nodito/vault.yml
|
|
# ------------------------------------------------------------------
|
|
- name: Refuse to run with a placeholder UPS password
|
|
assert:
|
|
that:
|
|
- ups_password is defined
|
|
- ups_password | length > 0
|
|
- ups_password != "CHANGE_ME_TO_SECURE_PASSWORD"
|
|
fail_msg: >-
|
|
ups_password is unset or still the placeholder. Applying this play would
|
|
overwrite the working /etc/nut/upsd.users and /etc/nut/upsmon.conf on
|
|
nodito and restart NUT. Put the real password in the vault first:
|
|
ansible-vault edit host_vars/nodito/vault.yml
|
|
|
|
# ------------------------------------------------------------------
|
|
# Installation
|
|
# ------------------------------------------------------------------
|
|
- name: Install NUT packages
|
|
apt:
|
|
name:
|
|
- nut
|
|
- nut-client
|
|
- nut-server
|
|
state: present
|
|
update_cache: true
|
|
|
|
# ------------------------------------------------------------------
|
|
# Verify UPS is detected
|
|
# ------------------------------------------------------------------
|
|
- name: Check if UPS is detected via USB
|
|
shell: lsusb | grep -i cyber
|
|
register: lsusb_output
|
|
changed_when: false
|
|
failed_when: false
|
|
|
|
- name: Display USB detection result
|
|
debug:
|
|
msg: "{{ lsusb_output.stdout | default('UPS not detected via USB - ensure it is plugged in') }}"
|
|
|
|
- name: Fail if UPS not detected
|
|
fail:
|
|
msg: "CyberPower UPS not detected via USB. Ensure the USB cable is connected."
|
|
when: lsusb_output.rc != 0
|
|
|
|
- name: Reload udev rules for USB permissions
|
|
shell: |
|
|
udevadm control --reload-rules
|
|
udevadm trigger --subsystem-match=usb --action=add
|
|
changed_when: true
|
|
|
|
- name: Verify USB device has nut group permissions
|
|
shell: |
|
|
BUS_DEV=$(lsusb | grep -i cyber | grep -oP 'Bus \K\d+|Device \K\d+' | tr '\n' '/' | sed 's/\/$//')
|
|
if [ -n "$BUS_DEV" ]; then
|
|
BUS=$(echo $BUS_DEV | cut -d'/' -f1)
|
|
DEV=$(echo $BUS_DEV | cut -d'/' -f2)
|
|
ls -la /dev/bus/usb/$BUS/$DEV
|
|
else
|
|
echo "UPS device not found"
|
|
exit 1
|
|
fi
|
|
register: usb_permissions
|
|
changed_when: false
|
|
|
|
- name: Display USB permissions
|
|
debug:
|
|
msg: "{{ usb_permissions.stdout }} (should show 'root nut', not 'root root')"
|
|
|
|
- name: Scan for UPS with nut-scanner
|
|
command: nut-scanner -U
|
|
register: nut_scanner_output
|
|
changed_when: false
|
|
failed_when: false
|
|
|
|
- name: Display nut-scanner result
|
|
debug:
|
|
msg: "{{ nut_scanner_output.stdout_lines }}"
|
|
|
|
# ------------------------------------------------------------------
|
|
# Configuration files
|
|
# ------------------------------------------------------------------
|
|
- name: Configure NUT mode (standalone)
|
|
template:
|
|
dest: /etc/nut/nut.conf
|
|
src: templates/nut.conf.j2
|
|
owner: root
|
|
group: nut
|
|
mode: "0640"
|
|
notify: Restart NUT services
|
|
|
|
- name: Configure UPS device
|
|
template:
|
|
dest: /etc/nut/ups.conf
|
|
src: templates/ups.conf.j2
|
|
owner: root
|
|
group: nut
|
|
mode: "0640"
|
|
notify: Restart NUT services
|
|
|
|
- name: Configure upsd to listen on localhost
|
|
template:
|
|
dest: /etc/nut/upsd.conf
|
|
src: templates/upsd.conf.j2
|
|
owner: root
|
|
group: nut
|
|
mode: "0640"
|
|
notify: Restart NUT services
|
|
|
|
- name: Configure upsd users
|
|
template:
|
|
dest: /etc/nut/upsd.users
|
|
src: templates/upsd.users.j2
|
|
owner: root
|
|
group: nut
|
|
mode: "0640"
|
|
notify: Restart NUT services
|
|
|
|
- name: Configure upsmon
|
|
template:
|
|
dest: /etc/nut/upsmon.conf
|
|
src: templates/upsmon.conf.j2
|
|
owner: root
|
|
group: nut
|
|
mode: "0640"
|
|
notify: Restart NUT services
|
|
|
|
# ------------------------------------------------------------------
|
|
# Verify late-stage shutdown script
|
|
# ------------------------------------------------------------------
|
|
- name: Verify nutshutdown script exists
|
|
stat:
|
|
path: /lib/systemd/system-shutdown/nutshutdown
|
|
register: nutshutdown_script
|
|
|
|
- name: Warn if nutshutdown script is missing
|
|
debug:
|
|
msg: "WARNING: /lib/systemd/system-shutdown/nutshutdown not found. UPS may not cut power after shutdown."
|
|
when: not nutshutdown_script.stat.exists
|
|
|
|
# ------------------------------------------------------------------
|
|
# Services
|
|
# ------------------------------------------------------------------
|
|
- name: Enable and start NUT driver enumerator
|
|
systemd:
|
|
name: nut-driver-enumerator
|
|
enabled: true
|
|
state: started
|
|
|
|
- name: Enable and start NUT server
|
|
systemd:
|
|
name: nut-server
|
|
enabled: true
|
|
state: started
|
|
|
|
- name: Enable and start NUT monitor
|
|
systemd:
|
|
name: nut-monitor
|
|
enabled: true
|
|
state: started
|
|
|
|
# ------------------------------------------------------------------
|
|
# Verification
|
|
# ------------------------------------------------------------------
|
|
- name: Wait for NUT services to stabilize
|
|
pause:
|
|
seconds: 3
|
|
|
|
- name: Verify NUT can communicate with UPS
|
|
command: upsc {{ ups_name }}@localhost
|
|
register: upsc_output
|
|
changed_when: false
|
|
failed_when: upsc_output.rc != 0
|
|
|
|
- name: Display UPS status
|
|
debug:
|
|
msg: "{{ upsc_output.stdout_lines }}"
|
|
|
|
- name: Get UPS status summary
|
|
shell: |
|
|
echo "Status: $(upsc {{ ups_name }}@localhost ups.status 2>/dev/null)"
|
|
echo "Battery: $(upsc {{ ups_name }}@localhost battery.charge 2>/dev/null)%"
|
|
echo "Runtime: $(upsc {{ ups_name }}@localhost battery.runtime 2>/dev/null)s"
|
|
echo "Load: $(upsc {{ ups_name }}@localhost ups.load 2>/dev/null)%"
|
|
register: ups_summary
|
|
changed_when: false
|
|
|
|
- name: Display UPS summary
|
|
debug:
|
|
msg: "{{ ups_summary.stdout_lines }}"
|
|
|
|
- name: Verify low battery thresholds
|
|
shell: |
|
|
echo "Runtime threshold: $(upsc {{ ups_name }}@localhost battery.runtime.low 2>/dev/null)s"
|
|
echo "Charge threshold: $(upsc {{ ups_name }}@localhost battery.charge.low 2>/dev/null)%"
|
|
register: thresholds
|
|
changed_when: false
|
|
|
|
- name: Display low battery thresholds
|
|
debug:
|
|
msg: "{{ thresholds.stdout_lines }}"
|
|
|
|
handlers:
|
|
- name: Restart NUT services
|
|
systemd:
|
|
name: "{{ item }}"
|
|
state: restarted
|
|
loop:
|
|
- nut-driver-enumerator
|
|
- nut-server
|
|
- nut-monitor
|
|
|
|
# The UPS heartbeat play that used to live here is gone. What it deployed -
|
|
# /opt/ups-monitoring plus a ups-heartbeat timer - is now the ups-status check
|
|
# in infra/400_host_monitoring.yml, which reports to Gatus like every other
|
|
# host check instead of carrying its own push plumbing.
|
|
#
|
|
# This playbook is now purely NUT setup: the driver, upsd, upsmon and the
|
|
# shutdown behaviour. Monitoring whether the UPS is on mains is a separate
|
|
# concern and belongs with the other host checks.
|