counterweight
3a9e1d5851
Three changes: detection windows tightened, a Signal transport deployed, and
alerts attached to all 84 non-transport endpoints.
── Failing faster ──────────────────────────────────────────────────────────
The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks
"did anything arrive in the last interval", so real detection is 1-2x the
window. And a window can only ever be as tight as the push frequency - which is
why two checks moved rather than just having their numbers changed.
liveness, cpu, ups, service-health, probes 16m -> 11m
disk, zfs daily/30h -> 6-hourly/7h
backup store + pull job daily/30h -> 6-hourly/7h
DNS records 24h -> 6h
backup dump 30h -> 26h
backup-dump stays slow because the dump genuinely is daily. The store-side check
catches the same fault within 6h by reading the source's dump timestamp out of
the artefact filename, so 26h is a backstop rather than the primary signal.
── Thresholds differ by check type, deliberately ───────────────────────────
`failure-threshold` counts consecutive failures, but "consecutive" is a
different amount of wall-clock time per check: a push endpoint produces one
failure per heartbeat window, a pulled one per interval. The default of 3 would
mean 33 minutes on an 11m heartbeat and over a day on a 7h one.
More importantly the heartbeat window ALREADY encodes the tolerance - an 11m
window on a 5-minute push is exactly "one missed push forgiven" - so stacking a
threshold of 3 triples a tolerance that was already chosen. Hence:
push/heartbeat endpoints failure-threshold 1
pulled, 5m (public) failure-threshold 3 (= 15 minutes)
pulled, 6h/24h (dns, domain) failure-threshold 1
── The Signal transport ────────────────────────────────────────────────────
roles/signal_api runs signal-cli-rest-api on the observability host, pinned by
digest, MODE=native.
It publishes NO PORTS. The API has no authentication of any kind - anything that
reaches it can send messages as you and read your Signal. Gatus talks to it over
a shared docker network by service name, which is also WHY the network exists:
Gatus runs in a container, so the host's loopback is unreachable from it and a
port published on 127.0.0.1 would not have worked.
MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs
Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary
per request, and alerts are rare enough that startup cost per alert is the right
trade.
Monitored - Gatus polls /v1/health over the same network path the alerts take,
so it proves the delivery route rather than mere container liveness. Deliberately
NOT backed up: the data directory holds Signal private keys and the recovery
path is to link the device again from the phone.
Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert.
If either is down, Signal is precisely what cannot deliver the alert.
── Four traps hit while linking, all now in the role README ────────────────
* /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while
the binary itself emits a perfectly good URI. Not worked around by switching
MODE, which would put a JVM in the path of every alert permanently.
* The data dir must be owned by uid 1000, not root. A root-owned 0700 dir
cannot be traversed by the container user, so linking silently never
completes and /v1/accounts returns "Failed to read local accounts list".
* `docker exec` runs as ROOT while the service runs as uid 1000, so without
--config the account is written to /root/... on the container's ephemeral
layer. It reports success and is destroyed on the next recreate.
* The phone reporting "network error" was IPv6: chat.signal.org resolves to
dualstack AAAA records first, the container has no IPv6 address, and this
host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed
with a mounted gai.conf that prefers IPv4.
Verified: provider loads (configuredProviders=[signal]), a test message was
delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
64 lines
2.4 KiB
YAML
64 lines
2.4 KiB
YAML
---
|
|
- name: Assert Docker is available
|
|
ansible.builtin.command: docker --version
|
|
register: gatus_docker_check
|
|
changed_when: false
|
|
failed_when: gatus_docker_check.rc != 0
|
|
|
|
# Created explicitly rather than by either compose file, so neither the gatus
|
|
# stack nor the signal-api stack has to be deployed before the other.
|
|
- name: Ensure the shared monitoring network exists
|
|
ansible.builtin.command: "docker network create {{ gatus_network }}"
|
|
register: gatus_net
|
|
changed_when: "'already exists' not in gatus_net.stderr"
|
|
failed_when:
|
|
- gatus_net.rc != 0
|
|
- "'already exists' not in gatus_net.stderr"
|
|
|
|
- name: Create the gatus directories
|
|
ansible.builtin.file:
|
|
path: "{{ item.path }}"
|
|
state: directory
|
|
owner: "{{ item.owner }}"
|
|
group: "{{ item.group }}"
|
|
mode: "{{ item.mode }}"
|
|
loop:
|
|
- {path: "{{ gatus_dir }}", owner: root, group: root, mode: "0755"}
|
|
# Config is root-owned and world-unreadable: it holds external-endpoint
|
|
# tokens. The container mounts it read-only and reads it as gatus_uid, so
|
|
# that id needs group access - hence the group ownership below.
|
|
- {path: "{{ gatus_config_dir }}", owner: root, group: "{{ gatus_gid }}", mode: "0750"}
|
|
- {path: "{{ gatus_endpoints_dir }}", owner: root, group: "{{ gatus_gid }}", mode: "0750"}
|
|
# Data is the one path the container writes to, so it must be owned by the
|
|
# numeric id the container runs as. `read_only: true` makes everything else
|
|
# in the container immutable.
|
|
- {path: "{{ gatus_data_dir }}", owner: "{{ gatus_uid }}", group: "{{ gatus_gid }}", mode: "0750"}
|
|
|
|
- name: Write the base gatus configuration
|
|
ansible.builtin.template:
|
|
src: config.yaml.j2
|
|
dest: "{{ gatus_config_dir }}/{{ gatus_base_config_file }}"
|
|
owner: root
|
|
group: "{{ gatus_gid }}"
|
|
mode: "0640"
|
|
notify: Restart gatus
|
|
|
|
# Without this the container crash-loops on an empty config. See the template.
|
|
- name: Write the gatus self-check endpoint
|
|
ansible.builtin.template:
|
|
src: endpoint-self.yaml.j2
|
|
dest: "{{ gatus_endpoints_dir }}/00-self.yaml"
|
|
owner: root
|
|
group: "{{ gatus_gid }}"
|
|
mode: "0640"
|
|
when: gatus_self_check | bool
|
|
notify: Restart gatus
|
|
|
|
- name: Write the docker compose file
|
|
ansible.builtin.template:
|
|
src: docker-compose.yml.j2
|
|
dest: "{{ gatus_dir }}/docker-compose.yml"
|
|
owner: root
|
|
group: root
|
|
mode: "0644"
|
|
notify: Restart gatus
|