A stopgap, committed so it is visible rather than a surprise: THE PHONE IS CURRENTLY OFF. Flip `enabled` back to true under the signal provider's default-alert to turn it on. Why one key reaches all ~90 endpoints: Gatus merges a provider's default-alert into each endpoint's alert for every field the endpoint left unset, and `enabled` is the only alert field no registration playbook sets. So this works without editing a single endpoint and without unregistering anything. The dashboard keeps checking and keeps showing red; it just stops sending. Deliberately NOT done by deleting the `signal:` provider block. That would strand ~90 endpoints referencing a provider that no longer exists, and throw away the number and the recipient list with it. The thing that made this necessary - a 5-minute poll turning one wedged service into 83 messages in six hours - is fixed properly in the preceding commit. This key is the blunt instrument that bought time to do that, and it should be turned back on once the cadence change has been observed for a few days. Known consequence while it is off: nothing pages. A six-day xmrblocks outage went unannounced precisely because of this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
121 lines
5.6 KiB
YAML
121 lines
5.6 KiB
YAML
---
|
|
# Gatus: health checks, status page and alerting for the whole estate.
|
|
#
|
|
# Built from source and run under systemd - upstream publishes no binaries, and
|
|
# their Dockerfile shows the runtime needs nothing but the static binary and a
|
|
# CA bundle. See roles/gatus/README.md.
|
|
- name: Deploy Gatus on the observability host
|
|
hosts: observability
|
|
become: yes
|
|
vars:
|
|
gatus_alerting:
|
|
signal:
|
|
# NOTE: the key is `api-url`, not `url` as upstream's own README table
|
|
# says - see alerting/provider/signal/signal.go. Gatus appends /v2/send
|
|
# itself if the suffix is missing.
|
|
#
|
|
# Reached by service name over the shared docker network. Gatus runs in
|
|
# a container, so 127.0.0.1 here would be the Gatus container, not the
|
|
# host - and the Signal API deliberately publishes no ports because it
|
|
# has no authentication.
|
|
api-url: "http://signal-api:8080"
|
|
number: "{{ signal_number }}"
|
|
recipients: "{{ signal_recipients }}"
|
|
default-alert:
|
|
# ── SIGNAL ALERTING IS OFF ──────────────────────────────────────
|
|
# Flip to true to turn the phone back on. Nothing else needs to
|
|
# change: this ONE key disables every Signal alert in the estate.
|
|
#
|
|
# It works because `enabled` is the only alert field no registration
|
|
# playbook sets. Gatus merges a provider's default-alert into each
|
|
# endpoint's alert for every field the endpoint left unset, so this
|
|
# reaches all ~90 of them without editing a single endpoint - and
|
|
# without unregistering anything. The dashboard keeps checking and
|
|
# keeps showing red; it just stops sending.
|
|
#
|
|
# Deliberately NOT done by deleting the `signal:` provider block:
|
|
# that would strand ~90 endpoints referencing a provider that no
|
|
# longer exists, and lose the number and recipients.
|
|
enabled: false
|
|
# Overridden per group by the registration playbooks; these are the
|
|
# values that apply if a caller sets nothing.
|
|
failure-threshold: 1
|
|
success-threshold: 2
|
|
send-on-resolved: true
|
|
# An ongoing outage should not become an ongoing phone buzz.
|
|
minimum-reminder-interval: 6h
|
|
|
|
roles:
|
|
- gatus
|
|
|
|
# The dashboard is bound to loopback; Caddy publishes it.
|
|
#
|
|
# Auth is done HERE, at the edge, and not with Gatus's own `security.basic`.
|
|
# Gatus's security middleware protects exactly four routes (api/api.go):
|
|
#
|
|
# /api/v1/endpoints/statuses
|
|
# /api/v1/endpoints/:key/statuses
|
|
# /api/v1/suites/statuses
|
|
# /api/v1/suites/:key/statuses
|
|
#
|
|
# Everything else is registered on the UNPROTECTED router, including
|
|
# /api/v1/config, every badge, and - the part that matters -
|
|
# /api/v1/endpoints/:key/uptimes/:duration and .../response-times/:duration/history,
|
|
# which return real per-endpoint data to anyone who can guess a key. Keys are
|
|
# just "<group>_<name>". So Gatus's own auth makes the dashboard render empty
|
|
# while leaving the data readable, which is worse than it looks.
|
|
#
|
|
# The one route that must NOT sit behind basic auth is the external-endpoint
|
|
# push API. It authenticates with `Authorization: Bearer <token>`, and basic
|
|
# auth wants `Authorization: Basic <...>` - same header, two schemes, and the
|
|
# push clients lose. It is not actually unauthenticated: the handler 401s on a
|
|
# missing prefix, an empty token, or a token that does not match that endpoint's
|
|
# own. Upstream's comment on the route says exactly that.
|
|
- name: Publish the Gatus status page through Caddy
|
|
hosts: observability
|
|
become: yes
|
|
tasks:
|
|
- name: Require the dashboard credentials to be set
|
|
ansible.builtin.assert:
|
|
that:
|
|
- gatus_dashboard_username is defined
|
|
- gatus_dashboard_username | length > 0
|
|
- gatus_dashboard_password_hash is defined
|
|
- gatus_dashboard_password_hash.startswith('$2')
|
|
fail_msg: >-
|
|
gatus_dashboard_username and gatus_dashboard_password_hash must be in
|
|
the vault. Generate the hash on the observability host, which runs
|
|
Caddy natively, so the bcrypt cost and format match what verifies it:
|
|
caddy hash-password --plaintext 'your-password'
|
|
then: ansible-vault edit group_vars/all/vault.yml
|
|
|
|
- name: Configure the Caddy vhost for Gatus
|
|
ansible.builtin.include_role:
|
|
name: caddy_site
|
|
vars:
|
|
caddy_site_name: gatus
|
|
caddy_site_domain: "{{ subdomains.gatus }}.{{ root_domain }}"
|
|
# caddy_site_body rather than caddy_site_upstream + caddy_site_basic_auth,
|
|
# because that pair applies auth to the whole site with no way to carve
|
|
# out the push path. `handle` blocks are mutually exclusive and first
|
|
# match wins, so the push API gets a route of its own.
|
|
caddy_site_body: |
|
|
@push {
|
|
path /api/v1/endpoints/*/external
|
|
method POST
|
|
}
|
|
|
|
# Push API: Bearer-authenticated by Gatus itself. No basic auth here,
|
|
# or the Authorization header collides.
|
|
handle @push {
|
|
reverse_proxy 127.0.0.1:{{ gatus_port | default(8080) }}
|
|
}
|
|
|
|
# Everything else: the dashboard, the config endpoint, the badges and
|
|
# the uptime/response-time history.
|
|
handle {
|
|
basic_auth {
|
|
{{ gatus_dashboard_username }} {{ gatus_dashboard_password_hash }}
|
|
}
|
|
reverse_proxy 127.0.0.1:{{ gatus_port | default(8080) }}
|
|
}
|