counterweight
853e62a19c
Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
90 lines
3.4 KiB
YAML
90 lines
3.4 KiB
YAML
---
|
|
- name: "Assert healthcheck '{{ healthcheck_name }}' is fully specified"
|
|
ansible.builtin.assert:
|
|
that:
|
|
- healthcheck_name | length > 0
|
|
- healthcheck_description | length > 0
|
|
- (healthcheck_check | length > 0) != (healthcheck_command | length > 0)
|
|
- not (healthcheck_push_url | length > 0) or (healthcheck_push_token | length > 0)
|
|
fail_msg: >-
|
|
healthcheck needs a name, a description, exactly one of healthcheck_check
|
|
or healthcheck_command, and a token whenever a push URL is set. A push URL
|
|
with no token would report to Gatus and be rejected 401 on every run.
|
|
|
|
# Deduplicated across the whole play run. This role is included once PER CHECK,
|
|
# and a host with several checks was otherwise running apt several times to
|
|
# install a curl that was already there - 29 apt transactions estate-wide, and
|
|
# the slowest thing in the deploy by a wide margin. The fact below remembers
|
|
# what has already been ensured on this host.
|
|
- name: Install healthcheck dependencies
|
|
ansible.builtin.package:
|
|
name: "{{ healthcheck_wanted_packages }}"
|
|
state: present
|
|
vars:
|
|
healthcheck_wanted_packages: >-
|
|
{{ (healthcheck_packages | default(['curl']))
|
|
| difference(healthcheck_installed_packages | default([])) }}
|
|
when: healthcheck_wanted_packages | length > 0
|
|
|
|
- name: Remember which dependencies this host already has
|
|
ansible.builtin.set_fact:
|
|
healthcheck_installed_packages: >-
|
|
{{ (healthcheck_installed_packages | default([]))
|
|
| union(healthcheck_packages | default(['curl'])) }}
|
|
|
|
- name: Create the healthcheck log directory
|
|
ansible.builtin.file:
|
|
path: "{{ healthcheck_log_dir }}"
|
|
state: directory
|
|
owner: root
|
|
group: root
|
|
mode: "0750"
|
|
|
|
- name: "Install the {{ healthcheck_name }} check script"
|
|
ansible.builtin.template:
|
|
src: healthcheck.sh.j2
|
|
dest: "{{ healthcheck_script_dir }}/{{ healthcheck_name }}-healthcheck.sh"
|
|
owner: root
|
|
group: root
|
|
mode: "0755"
|
|
|
|
# The token is in this unit file, so it must not be world-readable.
|
|
- name: "Install the {{ healthcheck_name }} systemd service"
|
|
ansible.builtin.template:
|
|
src: healthcheck.service.j2
|
|
dest: "/etc/systemd/system/{{ healthcheck_name }}-healthcheck.service"
|
|
owner: root
|
|
group: root
|
|
mode: "0600"
|
|
|
|
- name: "Install the {{ healthcheck_name }} systemd timer"
|
|
ansible.builtin.template:
|
|
src: healthcheck.timer.j2
|
|
dest: "/etc/systemd/system/{{ healthcheck_name }}-healthcheck.timer"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
|
|
# `restarted`, not `started`: started is a no-op on an already-active timer, so
|
|
# a changed interval or a stuck timer would never be picked up.
|
|
- name: "Enable and start the {{ healthcheck_name }} timer"
|
|
ansible.builtin.systemd:
|
|
name: "{{ healthcheck_name }}-healthcheck.timer"
|
|
enabled: yes
|
|
state: restarted
|
|
daemon_reload: yes
|
|
|
|
- name: "Run the {{ healthcheck_name }} check once now"
|
|
ansible.builtin.command: "{{ healthcheck_script_dir }}/{{ healthcheck_name }}-healthcheck.sh"
|
|
environment:
|
|
HEALTHCHECK_PUSH_URL: "{{ healthcheck_push_url }}"
|
|
HEALTHCHECK_PUSH_TOKEN: "{{ healthcheck_push_token }}"
|
|
register: healthcheck_first_run
|
|
changed_when: false
|
|
failed_when: false
|
|
|
|
- name: "Report the first {{ healthcheck_name }} result"
|
|
ansible.builtin.debug:
|
|
msg: >-
|
|
{{ healthcheck_name }}: {{ 'HEALTHY' if healthcheck_first_run.rc == 0
|
|
else 'UNHEALTHY (rc=' ~ healthcheck_first_run.rc ~ ')' }}
|