counterweight
3a9e1d5851
Three changes: detection windows tightened, a Signal transport deployed, and
alerts attached to all 84 non-transport endpoints.
── Failing faster ──────────────────────────────────────────────────────────
The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks
"did anything arrive in the last interval", so real detection is 1-2x the
window. And a window can only ever be as tight as the push frequency - which is
why two checks moved rather than just having their numbers changed.
liveness, cpu, ups, service-health, probes 16m -> 11m
disk, zfs daily/30h -> 6-hourly/7h
backup store + pull job daily/30h -> 6-hourly/7h
DNS records 24h -> 6h
backup dump 30h -> 26h
backup-dump stays slow because the dump genuinely is daily. The store-side check
catches the same fault within 6h by reading the source's dump timestamp out of
the artefact filename, so 26h is a backstop rather than the primary signal.
── Thresholds differ by check type, deliberately ───────────────────────────
`failure-threshold` counts consecutive failures, but "consecutive" is a
different amount of wall-clock time per check: a push endpoint produces one
failure per heartbeat window, a pulled one per interval. The default of 3 would
mean 33 minutes on an 11m heartbeat and over a day on a 7h one.
More importantly the heartbeat window ALREADY encodes the tolerance - an 11m
window on a 5-minute push is exactly "one missed push forgiven" - so stacking a
threshold of 3 triples a tolerance that was already chosen. Hence:
push/heartbeat endpoints failure-threshold 1
pulled, 5m (public) failure-threshold 3 (= 15 minutes)
pulled, 6h/24h (dns, domain) failure-threshold 1
── The Signal transport ────────────────────────────────────────────────────
roles/signal_api runs signal-cli-rest-api on the observability host, pinned by
digest, MODE=native.
It publishes NO PORTS. The API has no authentication of any kind - anything that
reaches it can send messages as you and read your Signal. Gatus talks to it over
a shared docker network by service name, which is also WHY the network exists:
Gatus runs in a container, so the host's loopback is unreachable from it and a
port published on 127.0.0.1 would not have worked.
MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs
Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary
per request, and alerts are rare enough that startup cost per alert is the right
trade.
Monitored - Gatus polls /v1/health over the same network path the alerts take,
so it proves the delivery route rather than mere container liveness. Deliberately
NOT backed up: the data directory holds Signal private keys and the recovery
path is to link the device again from the phone.
Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert.
If either is down, Signal is precisely what cannot deliver the alert.
── Four traps hit while linking, all now in the role README ────────────────
* /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while
the binary itself emits a perfectly good URI. Not worked around by switching
MODE, which would put a JVM in the path of every alert permanently.
* The data dir must be owned by uid 1000, not root. A root-owned 0700 dir
cannot be traversed by the container user, so linking silently never
completes and /v1/accounts returns "Failed to read local accounts list".
* `docker exec` runs as ROOT while the service runs as uid 1000, so without
--config the account is written to /root/... on the container's ephemeral
layer. It reports success and is destroyed on the next recreate.
* The phone reporting "network error" was IPv6: chat.signal.org resolves to
dualstack AAAA records first, the container has no IPv6 address, and this
host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed
with a mounted gai.conf that prefers IPv4.
Verified: provider loads (configuredProviders=[signal]), a test message was
delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
108 lines
6.5 KiB
YAML
108 lines
6.5 KiB
YAML
---
|
|
# Gatus, deployed the way upstream distributes it: the container image.
|
|
#
|
|
# Upstream publishes NO binary release assets - the image is the only artefact
|
|
# they ship, and therefore the only artefact they test. Building from source is
|
|
# possible (`go build` alone is enough; the Vue dashboard is compiled in via
|
|
# `//go:embed static`, and CGO_ENABLED=0 works because the sqlite driver is
|
|
# pure-Go modernc.org/sqlite) but it produces a binary upstream never ran, and
|
|
# it means a full compile of the AWS SDK and gRPC on the smallest box in the
|
|
# estate.
|
|
|
|
# ── Image ────────────────────────────────────────────────────────────────────
|
|
gatus_version: "v5.36.0"
|
|
# Pinned by DIGEST, not by tag. A tag is mutable - `v5.36.0` can be repushed -
|
|
# so pinning the tag alone is a weaker promise than it looks. The digest is the
|
|
# content address: if it resolves, it is byte-for-byte the image reviewed here.
|
|
# Both must be updated together; the tag is kept only so humans can read it.
|
|
gatus_image_digest: "sha256:c5f210d095fa78e6efaa20ffeb14803f2ba4f10615e16a6d12087697149617f0"
|
|
gatus_image: "ghcr.io/twin/gatus@{{ gatus_image_digest }}"
|
|
|
|
# ── Paths (host side) ────────────────────────────────────────────────────────
|
|
gatus_dir: /opt/gatus
|
|
gatus_config_dir: "{{ gatus_dir }}/config"
|
|
gatus_data_dir: "{{ gatus_dir }}/data"
|
|
|
|
# Gatus merges every *.yaml under GATUS_CONFIG_PATH and its subdirectories:
|
|
# maps deep-merge, lists append. That is why this role ships a config DIRECTORY
|
|
# rather than one file - each service contributes its own endpoint file, the
|
|
# same way each service contributes a vhost through `caddy_site`.
|
|
#
|
|
# Primitives must be defined exactly once across all files or the merge is
|
|
# ambiguous, so everything that is not a list lives in the base file and
|
|
# nowhere else.
|
|
gatus_base_config_file: "00-base.yaml"
|
|
gatus_endpoints_dir: "{{ gatus_config_dir }}/endpoints"
|
|
|
|
# ── Identity ─────────────────────────────────────────────────────────────────
|
|
# The image is FROM scratch, so it has no /etc/passwd and no user to drop to by
|
|
# name. Run it by numeric uid/gid instead, and own the data volume to match.
|
|
gatus_uid: 10001
|
|
gatus_gid: 10001
|
|
|
|
# ── Web ──────────────────────────────────────────────────────────────────────
|
|
gatus_port: 8080
|
|
# HOST-side address the container's port is published on. Gatus itself always
|
|
# binds 0.0.0.0 inside the container - see the note in config.yaml.j2. Never
|
|
# publish this on 0.0.0.0: the external-endpoint push API shares the dashboard's
|
|
# listener, and Caddy is what should be in front of both.
|
|
gatus_bind_address: "127.0.0.1"
|
|
gatus_ui_title: "Status"
|
|
gatus_ui_header: "Status"
|
|
|
|
# ── Storage ──────────────────────────────────────────────────────────────────
|
|
# sqlite, not memory: history has to survive a restart, or the dashboard lies
|
|
# about uptime after every deploy. Path is INSIDE the container.
|
|
gatus_storage_type: sqlite
|
|
gatus_storage_path: "/data/gatus.db"
|
|
gatus_storage_caching: true
|
|
|
|
# Per-endpoint row caps. Gatus bounds the database by COUNT, not by time, and
|
|
# trims inline on insert (storage/store/sql/sql.go, InsertEndpointResult) - so
|
|
# there is no retention job to write and no way for this to fill a disk.
|
|
#
|
|
# History depth is therefore a function of check frequency, not of days:
|
|
# 900 results is ~3 days of a 5-minute liveness check, and ~2.5 years of a daily
|
|
# disk check. Upstream's default is 100, which would have been 8 hours of
|
|
# liveness - not enough to still see a weekend incident on Monday.
|
|
#
|
|
# Note the uptime table is separate and its 30-day retention is hard-coded
|
|
# upstream (uptimeRetention), so uptime percentages top out at 30 days whatever
|
|
# this is set to.
|
|
gatus_storage_max_results: 900
|
|
gatus_storage_max_events: 50
|
|
|
|
# ── Alerting ─────────────────────────────────────────────────────────────────
|
|
# Pass-through: rendered verbatim under `alerting:`, so any provider Gatus
|
|
# supports works without touching this role. Empty means "check and record,
|
|
# alert nowhere" - valid, and the default until a provider is chosen.
|
|
gatus_alerting: {}
|
|
gatus_default_alerts: []
|
|
|
|
# ── Security ─────────────────────────────────────────────────────────────────
|
|
# gatus_basic_auth: {username: admin, password-bcrypt-base64: "..."}
|
|
gatus_basic_auth: {}
|
|
|
|
gatus_maintenance: {}
|
|
|
|
# Keep serving the previous config if a contributed endpoint file is malformed,
|
|
# instead of panicking. See the note in config.yaml.j2.
|
|
gatus_skip_invalid_config_update: true
|
|
gatus_log_level: INFO
|
|
|
|
# ── Self-check ───────────────────────────────────────────────────────────────
|
|
# Gatus panics on a config with no endpoints, so the role always ships one.
|
|
# Turning this off is only safe once another file in endpoints/ provides one.
|
|
gatus_self_check: true
|
|
|
|
# ── ICMP ─────────────────────────────────────────────────────────────────────
|
|
# Gatus supports icmp:// endpoints. Raw ICMP needs CAP_NET_RAW, which the
|
|
# container would get free only if it ran as root; it does not. Set false if
|
|
# you never use icmp:// checks and want the capability dropped entirely.
|
|
gatus_allow_icmp: true
|
|
|
|
# ── Shared network ───────────────────────────────────────────────────────────
|
|
# Gatus runs in a container, so the HOST's loopback is not reachable from it.
|
|
# Anything Gatus must talk to locally - the Signal API that sends its alerts -
|
|
# has to be on a shared docker network and addressed by service name.
|
|
gatus_network: monitoring
|