personal_infra/ansible/services/signal-api/deploy_signal_api_playbook.yml

45 lines
2 KiB
YAML
Raw Normal View History

alerting: Signal via signal-cli-rest-api, and faster failure detection Three changes: detection windows tightened, a Signal transport deployed, and alerts attached to all 84 non-transport endpoints. ── Failing faster ────────────────────────────────────────────────────────── The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks "did anything arrive in the last interval", so real detection is 1-2x the window. And a window can only ever be as tight as the push frequency - which is why two checks moved rather than just having their numbers changed. liveness, cpu, ups, service-health, probes 16m -> 11m disk, zfs daily/30h -> 6-hourly/7h backup store + pull job daily/30h -> 6-hourly/7h DNS records 24h -> 6h backup dump 30h -> 26h backup-dump stays slow because the dump genuinely is daily. The store-side check catches the same fault within 6h by reading the source's dump timestamp out of the artefact filename, so 26h is a backstop rather than the primary signal. ── Thresholds differ by check type, deliberately ─────────────────────────── `failure-threshold` counts consecutive failures, but "consecutive" is a different amount of wall-clock time per check: a push endpoint produces one failure per heartbeat window, a pulled one per interval. The default of 3 would mean 33 minutes on an 11m heartbeat and over a day on a 7h one. More importantly the heartbeat window ALREADY encodes the tolerance - an 11m window on a 5-minute push is exactly "one missed push forgiven" - so stacking a threshold of 3 triples a tolerance that was already chosen. Hence: push/heartbeat endpoints failure-threshold 1 pulled, 5m (public) failure-threshold 3 (= 15 minutes) pulled, 6h/24h (dns, domain) failure-threshold 1 ── The Signal transport ──────────────────────────────────────────────────── roles/signal_api runs signal-cli-rest-api on the observability host, pinned by digest, MODE=native. It publishes NO PORTS. The API has no authentication of any kind - anything that reaches it can send messages as you and read your Signal. Gatus talks to it over a shared docker network by service name, which is also WHY the network exists: Gatus runs in a container, so the host's loopback is unreachable from it and a port published on 127.0.0.1 would not have worked. MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary per request, and alerts are rare enough that startup cost per alert is the right trade. Monitored - Gatus polls /v1/health over the same network path the alerts take, so it proves the delivery route rather than mere container liveness. Deliberately NOT backed up: the data directory holds Signal private keys and the recovery path is to link the device again from the phone. Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert. If either is down, Signal is precisely what cannot deliver the alert. ── Four traps hit while linking, all now in the role README ──────────────── * /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while the binary itself emits a perfectly good URI. Not worked around by switching MODE, which would put a JVM in the path of every alert permanently. * The data dir must be owned by uid 1000, not root. A root-owned 0700 dir cannot be traversed by the container user, so linking silently never completes and /v1/accounts returns "Failed to read local accounts list". * `docker exec` runs as ROOT while the service runs as uid 1000, so without --config the account is written to /root/... on the container's ephemeral layer. It reports success and is destroyed on the next recreate. * The phone reporting "network error" was IPv6: chat.signal.org resolves to dualstack AAAA records first, the container has no IPv6 address, and this host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed with a mounted gai.conf that prefers IPv4. Verified: provider loads (configuredProviders=[signal]), a test message was delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 21:40:37 +02:00
---
# The Signal transport for Gatus alerts.
#
# Deliberately NOT published and NOT fronted by Caddy: the API has no
# authentication of any kind, so it is reachable only from the shared docker
# network that Gatus is on. See roles/signal_api/README.md, including the
# one-time manual step to link the device.
- name: Deploy the Signal API on the observability host
hosts: observability
become: yes
roles:
- signal_api
# post_tasks, not a second play: the registration below needs the role's
# defaults (service name, port) in scope, and a separate play would not have
# them.
post_tasks:
# Monitored, because a dead alert transport is the worst kind of dead: every
# check could be failing and nothing would tell you. Gatus polls it over the
# shared network - the same path the alerts take - so this proves the actual
# delivery route rather than merely that a container is running.
#
# Deliberately NOT backed up: the data directory holds Signal private keys,
# and the recovery path is to link the device again from the phone. Backing
# it up would copy a credential off the host to buy nothing.
- name: Register the signal-api health endpoint with Gatus
ansible.builtin.include_role:
name: gatus_endpoint
vars:
gatus_endpoint_name: signal-api
gatus_endpoint_pulled:
- name: signal-api
group: infrastructure
url: "http://{{ signal_api_service_name }}:{{ signal_api_port }}/v1/health"
interval: 5m
# Deliberately NOT alerted via Signal: if this endpoint is down,
# Signal is exactly what cannot deliver the alert. It is visible on
# the dashboard, and its failure shows up indirectly as every other
# alert going missing.
conditions:
# /v1/health answers 204 No Content, not 200 - checked live. Any
# 2xx is asserted rather than the exact code, so an upstream
# change from 204 to 200 does not read as an outage.
- "[STATUS] < 300"