personal_infra/ansible/roles/signal_api/README.md

134 lines
5.3 KiB
Markdown
Raw Normal View History

alerting: Signal via signal-cli-rest-api, and faster failure detection Three changes: detection windows tightened, a Signal transport deployed, and alerts attached to all 84 non-transport endpoints. ── Failing faster ────────────────────────────────────────────────────────── The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks "did anything arrive in the last interval", so real detection is 1-2x the window. And a window can only ever be as tight as the push frequency - which is why two checks moved rather than just having their numbers changed. liveness, cpu, ups, service-health, probes 16m -> 11m disk, zfs daily/30h -> 6-hourly/7h backup store + pull job daily/30h -> 6-hourly/7h DNS records 24h -> 6h backup dump 30h -> 26h backup-dump stays slow because the dump genuinely is daily. The store-side check catches the same fault within 6h by reading the source's dump timestamp out of the artefact filename, so 26h is a backstop rather than the primary signal. ── Thresholds differ by check type, deliberately ─────────────────────────── `failure-threshold` counts consecutive failures, but "consecutive" is a different amount of wall-clock time per check: a push endpoint produces one failure per heartbeat window, a pulled one per interval. The default of 3 would mean 33 minutes on an 11m heartbeat and over a day on a 7h one. More importantly the heartbeat window ALREADY encodes the tolerance - an 11m window on a 5-minute push is exactly "one missed push forgiven" - so stacking a threshold of 3 triples a tolerance that was already chosen. Hence: push/heartbeat endpoints failure-threshold 1 pulled, 5m (public) failure-threshold 3 (= 15 minutes) pulled, 6h/24h (dns, domain) failure-threshold 1 ── The Signal transport ──────────────────────────────────────────────────── roles/signal_api runs signal-cli-rest-api on the observability host, pinned by digest, MODE=native. It publishes NO PORTS. The API has no authentication of any kind - anything that reaches it can send messages as you and read your Signal. Gatus talks to it over a shared docker network by service name, which is also WHY the network exists: Gatus runs in a container, so the host's loopback is unreachable from it and a port published on 127.0.0.1 would not have worked. MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary per request, and alerts are rare enough that startup cost per alert is the right trade. Monitored - Gatus polls /v1/health over the same network path the alerts take, so it proves the delivery route rather than mere container liveness. Deliberately NOT backed up: the data directory holds Signal private keys and the recovery path is to link the device again from the phone. Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert. If either is down, Signal is precisely what cannot deliver the alert. ── Four traps hit while linking, all now in the role README ──────────────── * /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while the binary itself emits a perfectly good URI. Not worked around by switching MODE, which would put a JVM in the path of every alert permanently. * The data dir must be owned by uid 1000, not root. A root-owned 0700 dir cannot be traversed by the container user, so linking silently never completes and /v1/accounts returns "Failed to read local accounts list". * `docker exec` runs as ROOT while the service runs as uid 1000, so without --config the account is written to /root/... on the container's ephemeral layer. It reports success and is destroyed on the next recreate. * The phone reporting "network error" was IPv6: chat.signal.org resolves to dualstack AAAA records first, the container has no IPv6 address, and this host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed with a mounted gai.conf that prefers IPv4. Verified: provider loads (configuredProviders=[signal]), a test message was delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 21:40:37 +02:00
# signal_api
Runs [signal-cli-rest-api](https://github.com/bbernhard/signal-cli-rest-api) on
the `observability` host. Gatus uses it to deliver alerts over Signal.
Gatus does not speak Signal — it POSTs JSON to this service, which holds the
Signal identity and does the protocol work.
## It is never published, and that is not optional
**This API has no authentication of any kind.** No key, no token, no basic auth.
Anything that can reach the port can send messages as your identity and read
your Signal. So the compose file publishes **no ports at all** and there is no
Caddy vhost.
Gatus reaches it over a shared docker network (`monitoring`) by service name:
`http://signal-api:8080`. That is also *why* a shared network is needed rather
than a published port — Gatus runs in a container, so `127.0.0.1` for Gatus is
the Gatus container, not the host.
The network is created by an explicit Ansible task in both this role and
`gatus`, so neither stack has to be deployed before the other.
## MODE, and why `native`
Upstream offers `normal`, `native`, `json-rpc` and `json-rpc-native`. The
json-rpc modes keep a resident JVM daemon and upstream describes them as
"increased memory".
**This VPS has 464 MB of RAM**, already running Gatus and Caddy. A resident JVM
is not affordable. `native` runs a precompiled GraalVM binary per request — no
daemon, no resident cost — and alerts are rare enough that paying startup cost
per alert is the right trade.
## Linking the device — a one-time manual step
Ansible cannot scan a QR code, so this is manual. **Do not use
`/v1/qrcodelink`** — it is broken in `native` mode.
### The trap
`GET /v1/qrcodelink?device_name=...` returns:
```json
{"error":"Couldn't create QR code: no data to encode"}
```
The linking itself is fine: running the binary directly inside the container
emits a perfectly good provisioning URI.
```
$ docker exec signal-api signal-cli-native link -n gatus
sgnl://linkdevice?uuid=...&pub_key=...
```
It is the REST wrapper that fails to capture that output in `native` mode.
**Do not "fix" this by switching MODE to `normal` or `json-rpc`.** That puts a
JVM in the path of *every alert* on a 464 MB host, permanently degrading the
running system to work around a step performed once. Generate the QR yourself
instead.
### The procedure
**`docker exec` runs as root, but the service runs as uid 1000.** Without
`--config`, signal-cli writes the linked account to `/root/.local/share/signal-cli`
— the container's ephemeral layer, NOT the mounted volume. It looks like it
worked (`Associated with: +34…`), `/v1/accounts` keeps returning `[]`, and the
account is destroyed on the next `docker compose up`. Always pass `--config`.
1. Start the link and capture the URI. It must keep running while you scan:
docker exec signal-api sh -c "rm -f /tmp/link.uri; \
nohup signal-cli-native --config /home/.local/share/signal-cli \
link -n gatus > /tmp/link.uri 2>/tmp/link.log & echo started"
sleep 10
docker exec signal-api cat /tmp/link.uri
Do **not** add `setsid`, and do **not** background `docker exec` itself from
the host — the first stops the URI appearing, the second is killed when the
Ansible task returns. The output is block-buffered because stdout is a file,
so the URI appears only after several seconds; `stdbuf` does not help, as the
buffering is GraalVM's, not libc's.
2. Render the QR on your own machine and scan it:
qrencode -o /tmp/qr.png -s 12 -m 4 "sgnl://linkdevice?uuid=...&pub_key=..."
3. Phone: Signal → Settings → Linked devices → **+** → scan. Provisioning links
expire in a couple of minutes, so generate and scan in one sitting.
4. Confirm — this must list the number, not `[]`:
docker exec signal-api curl -s http://localhost:8080/v1/accounts
5. Send a test message:
docker exec signal-api curl -s -X POST -H "Content-Type: application/json" \
-d '{"message":"test","number":"+34…","recipients":["+34…"]}' \
http://localhost:8080/v2/send
Alerts are sent **from your own number**, so sending to yourself lands in Note
to Self. If the device is ever unlinked from the phone, alerts stop silently —
which is why this service is itself monitored.
### If the phone says "network error"
The phone is not the problem. `chat.signal.org` resolves to AWS Global
Accelerator **dualstack** addresses with the AAAA records first, this container
has no IPv6 address at all, and this host's IPv6 path is broken — the same edge
that returned a bogus 404 for the Go tarball. signal-cli reaches for an
unreachable IPv6 address and dies with `Link request error: Connection closed!`,
while the phone can only report a failed handshake.
That is what `gai.conf` (mounted at `/etc/gai.conf`) fixes. If linking starts
failing again, check it is still mounted and that `getent ahosts chat.signal.org`
returns an IPv4 address first.
## Backups
Deliberately **not** backed up. The data directory holds Signal private keys,
and the recovery path is to link again from the phone — which takes a minute and
does not depend on any stored artefact. Backing it up would copy a credential
off the host to buy nothing.
## Verifying
```bash
docker ps --filter name=signal-api
docker exec signal-api curl -fsS http://localhost:8080/v1/health
docker exec signal-api curl -fsS http://localhost:8080/v1/accounts
docker logs signal-api --tail 50
```