counterweight
853e62a19c
Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
109 lines
4.2 KiB
YAML
109 lines
4.2 KiB
YAML
---
|
|
# Remove the Uptime-Kuma-era monitoring that 400/401/402 replaced.
|
|
#
|
|
# Deleting the playbooks that installed these is NOT enough: the units are on
|
|
# the hosts, enabled, and keep firing regardless of what the repo says. Two of
|
|
# them still push to https://uptime.contrapeso.xyz every 15 minutes. A playbook
|
|
# that is deleted without a cleanup leaves its output running forever, with
|
|
# nothing in the repo left to explain it.
|
|
#
|
|
# What replaced what, all verified against the deployed scripts before removal:
|
|
#
|
|
# disk-usage-monitor -> disk-usage-healthcheck (infra/400)
|
|
# The old one checked ONLY "/" at 80%. The replacement walks every real
|
|
# filesystem, excluding tmpfs/devtmpfs/squashfs/overlay, at 85%. Strictly
|
|
# more coverage, so nothing is lost.
|
|
#
|
|
# system-healthcheck -> liveness-healthcheck (infra/400)
|
|
# The old script computed uptime and pushed. That is exactly a liveness
|
|
# heartbeat and nothing more.
|
|
#
|
|
# nodito-cpu-temp-monitor -> cpu-temp-healthcheck (infra/400)
|
|
# zfs-health-monitor -> zfs-health-healthcheck (infra/400)
|
|
# The ZFS check logic was ported verbatim - same five conditions - so only
|
|
# the reporting transport changed.
|
|
#
|
|
# NOT removed, because they are not monitoring:
|
|
# zfs-monthly-scrub.{timer,service} the actual scrub (infra/nodito/32)
|
|
# pull-backups, check-backups the backup machinery (playbooks/backups)
|
|
#
|
|
# This play is idempotent and kept permanently rather than run once and deleted:
|
|
# on a host that never had these it does nothing, and it guarantees a rebuilt or
|
|
# restored machine cannot quietly bring them back.
|
|
|
|
- name: Remove the legacy Uptime Kuma monitoring units
|
|
hosts: managed
|
|
become: yes
|
|
|
|
vars:
|
|
legacy_units:
|
|
- disk-usage-monitor
|
|
- system-healthcheck
|
|
- nodito-cpu-temp-monitor
|
|
- zfs-health-monitor
|
|
legacy_dirs:
|
|
- /opt/disk-monitoring
|
|
- /opt/system-healthcheck
|
|
- /opt/nodito-monitoring
|
|
- /opt/zfs-monitoring
|
|
|
|
tasks:
|
|
- name: Find which legacy units exist here
|
|
ansible.builtin.stat:
|
|
path: "/etc/systemd/system/{{ item.0 }}.{{ item.1 }}"
|
|
register: legacy_unit_files
|
|
loop: "{{ legacy_units | product(['timer', 'service']) | list }}"
|
|
|
|
# Stop and disable BEFORE deleting the unit file: systemd cannot disable a
|
|
# unit whose file has already gone, which would leave a dangling symlink in
|
|
# multi-user.target.wants and a warning on every daemon-reload.
|
|
- name: Stop and disable the legacy units
|
|
ansible.builtin.systemd:
|
|
name: "{{ item.item.0 }}.{{ item.item.1 }}"
|
|
state: stopped
|
|
enabled: no
|
|
loop: "{{ legacy_unit_files.results }}"
|
|
loop_control:
|
|
label: "{{ item.item.0 }}.{{ item.item.1 }}"
|
|
when: item.stat.exists
|
|
failed_when: false
|
|
|
|
- name: Remove the legacy unit files
|
|
ansible.builtin.file:
|
|
path: "/etc/systemd/system/{{ item.item.0 }}.{{ item.item.1 }}"
|
|
state: absent
|
|
loop: "{{ legacy_unit_files.results }}"
|
|
loop_control:
|
|
label: "{{ item.item.0 }}.{{ item.item.1 }}"
|
|
when: item.stat.exists
|
|
|
|
- name: Reload systemd
|
|
ansible.builtin.systemd:
|
|
daemon_reload: yes
|
|
|
|
- name: Remove the legacy monitoring scripts and their logs
|
|
ansible.builtin.file:
|
|
path: "{{ item }}"
|
|
state: absent
|
|
loop: "{{ legacy_dirs }}"
|
|
|
|
# An orphan predating all of this: mode 0644, not executable, referenced by
|
|
# no unit and no cron entry, pushing to a Kuma monitor. Superseded by
|
|
# ups-status-healthcheck.
|
|
- name: Remove the orphaned hand-written UPS heartbeat
|
|
ansible.builtin.file:
|
|
path: /usr/local/bin/ups-heartbeat.sh
|
|
state: absent
|
|
|
|
- name: Confirm nothing still pushes to Uptime Kuma
|
|
ansible.builtin.shell: >-
|
|
grep -rl "uptime.contrapeso.xyz" /etc/systemd/system /usr/local/bin /opt 2>/dev/null || true
|
|
register: kuma_refs
|
|
changed_when: false
|
|
|
|
- name: Report any remaining references
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
{{ 'clean - nothing references Uptime Kuma'
|
|
if kuma_refs.stdout | trim | length == 0
|
|
else 'STILL REFERENCING KUMA: ' ~ kuma_refs.stdout_lines | join(', ') }}
|