monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity
Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
99760dfd46
commit
853e62a19c
GPG key ID:
883EDBAA726BD96C
16 changed files with 184 additions and 1520 deletions
|
|
@ -170,141 +170,55 @@
|
|||
when: "'ONLINE' not in final_zfs_status.stdout"
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# ZFS health monitoring and monthly scrub.
|
||||
# The monthly scrub.
|
||||
#
|
||||
# The check script decides healthy/unhealthy and says so in its exit code, which
|
||||
# systemd keeps: systemctl is-failed zfs-health-monitor.service
|
||||
# The ZFS HEALTH CHECK that used to share this play is gone: it is now the
|
||||
# zfs-health check in infra/400_host_monitoring.yml, which carries the same five
|
||||
# conditions - pool state, device states, resilver in progress, read/write/
|
||||
# checksum errors, and errors from the last scan - but reports to Gatus like
|
||||
# every other host check instead of owning its own push plumbing.
|
||||
#
|
||||
# Reporting anywhere else is optional. Set `healthcheck_push_url` and the script
|
||||
# will also GET it with ?status=up|down; leave it empty and the exit code is
|
||||
# still the whole answer. Today that URL points at Uptime Kuma, which is still
|
||||
# running on watchtower but is no longer deployed by Ansible - the credentials
|
||||
# were retired, the service was not. If it is ever replaced, `healthcheck_push_url`
|
||||
# is the only thing that needs to change here.
|
||||
# The scrub itself stays here, because it is not monitoring: it is the
|
||||
# maintenance that gives the health check something true to report. A pool that
|
||||
# is never scrubbed has no idea whether it is healthy.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
- name: Setup ZFS Pool Health Monitoring and Monthly Scrubs
|
||||
- name: Schedule the monthly ZFS scrub
|
||||
hosts: hypervisor
|
||||
become: true
|
||||
vars_files:
|
||||
- ../../infra_vars.yml
|
||||
|
||||
vars:
|
||||
zfs_check_interval_seconds: 86400 # 24 hours
|
||||
zfs_check_timeout_seconds: 90000 # ~25 hours (interval + buffer)
|
||||
zfs_check_retries: 1
|
||||
zfs_monitoring_script_dir: /opt/zfs-monitoring
|
||||
zfs_monitoring_script_path: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.sh"
|
||||
zfs_log_file: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.log"
|
||||
zfs_systemd_health_service_name: zfs-health-monitor
|
||||
zfs_systemd_scrub_service_name: zfs-monthly-scrub
|
||||
# Optional. Empty is fine and is not an error - see the banner above.
|
||||
healthcheck_push_url: "{{ healthcheck_push_urls.zfs_health | default('') }}"
|
||||
|
||||
tasks:
|
||||
- name: Install required packages for ZFS monitoring
|
||||
package:
|
||||
name:
|
||||
- curl
|
||||
- jq
|
||||
state: present
|
||||
|
||||
- name: Create monitoring script directory
|
||||
file:
|
||||
path: "{{ zfs_monitoring_script_dir }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0755'
|
||||
|
||||
- name: Create ZFS health monitoring script
|
||||
template:
|
||||
dest: "{{ zfs_monitoring_script_path }}"
|
||||
src: templates/zfs_health_monitor.sh.j2
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0755'
|
||||
|
||||
- name: Create systemd service for ZFS health monitoring
|
||||
template:
|
||||
dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.service"
|
||||
src: templates/zfs-health-monitor.service.j2
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0644'
|
||||
|
||||
- name: Create systemd timer for daily ZFS health monitoring
|
||||
template:
|
||||
dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.timer"
|
||||
src: templates/zfs-health-monitor.timer.j2
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0644'
|
||||
|
||||
- name: Create systemd service for ZFS monthly scrub
|
||||
template:
|
||||
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.service"
|
||||
src: templates/zfs-monthly-scrub.service.j2
|
||||
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.service"
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0644'
|
||||
|
||||
- name: Create systemd timer for monthly ZFS scrub
|
||||
template:
|
||||
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.timer"
|
||||
src: templates/zfs-monthly-scrub.timer.j2
|
||||
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.timer"
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0644'
|
||||
|
||||
- name: Reload systemd daemon
|
||||
systemd:
|
||||
daemon_reload: yes
|
||||
|
||||
- name: Enable and start ZFS health monitoring timer
|
||||
systemd:
|
||||
name: "{{ zfs_systemd_health_service_name }}.timer"
|
||||
enabled: yes
|
||||
state: started
|
||||
|
||||
- name: Enable and start ZFS monthly scrub timer
|
||||
- name: Enable and start the monthly scrub timer
|
||||
systemd:
|
||||
name: "{{ zfs_systemd_scrub_service_name }}.timer"
|
||||
enabled: yes
|
||||
state: started
|
||||
daemon_reload: yes
|
||||
|
||||
- name: Test ZFS health monitoring script
|
||||
command: "{{ zfs_monitoring_script_path }}"
|
||||
register: script_test
|
||||
changed_when: false
|
||||
|
||||
- name: Verify script execution
|
||||
assert:
|
||||
that:
|
||||
- script_test.rc == 0
|
||||
fail_msg: "ZFS health monitoring script failed - check pool health"
|
||||
|
||||
- name: Display monitoring configuration
|
||||
- name: Report the scrub schedule
|
||||
debug:
|
||||
msg: |
|
||||
✓ ZFS Pool Health Monitoring deployed successfully!
|
||||
|
||||
Pool Name: {{ zfs_pool_name }}
|
||||
Push URL: {{ healthcheck_push_url | default('', true) | ternary('set', 'not set - exit code only') }}
|
||||
|
||||
Health Check:
|
||||
- Frequency: Every {{ zfs_check_interval_seconds }} seconds (24 hours)
|
||||
- Timeout: {{ zfs_check_timeout_seconds }} seconds (~25 hours)
|
||||
- Script: {{ zfs_monitoring_script_path }}
|
||||
- Log: {{ zfs_log_file }}
|
||||
- Service: {{ zfs_systemd_health_service_name }}.service
|
||||
- Timer: {{ zfs_systemd_health_service_name }}.timer
|
||||
|
||||
Monthly Scrub:
|
||||
- Schedule: Last day of month at 4:00 AM
|
||||
- Service: {{ zfs_systemd_scrub_service_name }}.service
|
||||
- Timer: {{ zfs_systemd_scrub_service_name }}.timer
|
||||
|
||||
Conditions monitored:
|
||||
- Pool state (must be ONLINE)
|
||||
- Device states (no DEGRADED/FAULTED/OFFLINE/UNAVAIL)
|
||||
- Resilver status (alerts if resilvering)
|
||||
- Read/Write/Checksum errors
|
||||
- Scrub errors
|
||||
msg: >-
|
||||
Monthly scrub of {{ zfs_pool_name }}:
|
||||
last day of each month at 04:00.
|
||||
Health is reported separately by the zfs-health check
|
||||
(infra/400_host_monitoring.yml).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue