alerting: Signal via signal-cli-rest-api, and faster failure detection

Three changes: detection windows tightened, a Signal transport deployed, and
alerts attached to all 84 non-transport endpoints.

── Failing faster ──────────────────────────────────────────────────────────
The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks
"did anything arrive in the last interval", so real detection is 1-2x the
window. And a window can only ever be as tight as the push frequency - which is
why two checks moved rather than just having their numbers changed.

  liveness, cpu, ups, service-health, probes   16m -> 11m
  disk, zfs                       daily/30h -> 6-hourly/7h
  backup store + pull job         daily/30h -> 6-hourly/7h
  DNS records                           24h -> 6h
  backup dump                             30h -> 26h

backup-dump stays slow because the dump genuinely is daily. The store-side check
catches the same fault within 6h by reading the source's dump timestamp out of
the artefact filename, so 26h is a backstop rather than the primary signal.

── Thresholds differ by check type, deliberately ───────────────────────────
`failure-threshold` counts consecutive failures, but "consecutive" is a
different amount of wall-clock time per check: a push endpoint produces one
failure per heartbeat window, a pulled one per interval. The default of 3 would
mean 33 minutes on an 11m heartbeat and over a day on a 7h one.

More importantly the heartbeat window ALREADY encodes the tolerance - an 11m
window on a 5-minute push is exactly "one missed push forgiven" - so stacking a
threshold of 3 triples a tolerance that was already chosen. Hence:

  push/heartbeat endpoints   failure-threshold 1
  pulled, 5m (public)        failure-threshold 3   (= 15 minutes)
  pulled, 6h/24h (dns, domain) failure-threshold 1

── The Signal transport ────────────────────────────────────────────────────
roles/signal_api runs signal-cli-rest-api on the observability host, pinned by
digest, MODE=native.

It publishes NO PORTS. The API has no authentication of any kind - anything that
reaches it can send messages as you and read your Signal. Gatus talks to it over
a shared docker network by service name, which is also WHY the network exists:
Gatus runs in a container, so the host's loopback is unreachable from it and a
port published on 127.0.0.1 would not have worked.

MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs
Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary
per request, and alerts are rare enough that startup cost per alert is the right
trade.

Monitored - Gatus polls /v1/health over the same network path the alerts take,
so it proves the delivery route rather than mere container liveness. Deliberately
NOT backed up: the data directory holds Signal private keys and the recovery
path is to link the device again from the phone.

Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert.
If either is down, Signal is precisely what cannot deliver the alert.

── Four traps hit while linking, all now in the role README ────────────────
  * /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while
    the binary itself emits a perfectly good URI. Not worked around by switching
    MODE, which would put a JVM in the path of every alert permanently.
  * The data dir must be owned by uid 1000, not root. A root-owned 0700 dir
    cannot be traversed by the container user, so linking silently never
    completes and /v1/accounts returns "Failed to read local accounts list".
  * `docker exec` runs as ROOT while the service runs as uid 1000, so without
    --config the account is written to /root/... on the container's ephemeral
    layer. It reports success and is destroyed on the next recreate.
  * The phone reporting "network error" was IPv6: chat.signal.org resolves to
    dualstack AAAA records first, the container has no IPv6 address, and this
    host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed
    with a mounted gai.conf that prefers IPv4.

Verified: provider loads (configuredProviders=[signal]), a test message was
delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/signal_api/README.md

@@ -0,0 +1,133 @@
+# signal_api
+
+Runs [signal-cli-rest-api](https://github.com/bbernhard/signal-cli-rest-api) on
+the `observability` host. Gatus uses it to deliver alerts over Signal.
+
+Gatus does not speak Signal — it POSTs JSON to this service, which holds the
+Signal identity and does the protocol work.
+
+## It is never published, and that is not optional
+
+**This API has no authentication of any kind.** No key, no token, no basic auth.
+Anything that can reach the port can send messages as your identity and read
+your Signal. So the compose file publishes **no ports at all** and there is no
+Caddy vhost.
+
+Gatus reaches it over a shared docker network (`monitoring`) by service name:
+`http://signal-api:8080`. That is also *why* a shared network is needed rather
+than a published port — Gatus runs in a container, so `127.0.0.1` for Gatus is
+the Gatus container, not the host.
+
+The network is created by an explicit Ansible task in both this role and
+`gatus`, so neither stack has to be deployed before the other.
+
+## MODE, and why `native`
+
+Upstream offers `normal`, `native`, `json-rpc` and `json-rpc-native`. The
+json-rpc modes keep a resident JVM daemon and upstream describes them as
+"increased memory".
+
+**This VPS has 464 MB of RAM**, already running Gatus and Caddy. A resident JVM
+is not affordable. `native` runs a precompiled GraalVM binary per request — no
+daemon, no resident cost — and alerts are rare enough that paying startup cost
+per alert is the right trade.
+
+## Linking the device — a one-time manual step
+
+Ansible cannot scan a QR code, so this is manual. **Do not use
+`/v1/qrcodelink`** — it is broken in `native` mode.
+
+### The trap
+
+`GET /v1/qrcodelink?device_name=...` returns:
+
+```json
+{"error":"Couldn't create QR code: no data to encode"}
+```
+
+The linking itself is fine: running the binary directly inside the container
+emits a perfectly good provisioning URI.
+
+```
+$ docker exec signal-api signal-cli-native link -n gatus
+sgnl://linkdevice?uuid=...&pub_key=...
+```
+
+It is the REST wrapper that fails to capture that output in `native` mode.
+
+**Do not "fix" this by switching MODE to `normal` or `json-rpc`.** That puts a
+JVM in the path of *every alert* on a 464 MB host, permanently degrading the
+running system to work around a step performed once. Generate the QR yourself
+instead.
+
+### The procedure
+
+**`docker exec` runs as root, but the service runs as uid 1000.** Without
+`--config`, signal-cli writes the linked account to `/root/.local/share/signal-cli`
+— the container's ephemeral layer, NOT the mounted volume. It looks like it
+worked (`Associated with: +34…`), `/v1/accounts` keeps returning `[]`, and the
+account is destroyed on the next `docker compose up`. Always pass `--config`.
+
+1. Start the link and capture the URI. It must keep running while you scan:
+
+       docker exec signal-api sh -c "rm -f /tmp/link.uri; \
+         nohup signal-cli-native --config /home/.local/share/signal-cli \
+           link -n gatus > /tmp/link.uri 2>/tmp/link.log & echo started"
+       sleep 10
+       docker exec signal-api cat /tmp/link.uri
+
+   Do **not** add `setsid`, and do **not** background `docker exec` itself from
+   the host — the first stops the URI appearing, the second is killed when the
+   Ansible task returns. The output is block-buffered because stdout is a file,
+   so the URI appears only after several seconds; `stdbuf` does not help, as the
+   buffering is GraalVM's, not libc's.
+
+2. Render the QR on your own machine and scan it:
+
+       qrencode -o /tmp/qr.png -s 12 -m 4 "sgnl://linkdevice?uuid=...&pub_key=..."
+
+3. Phone: Signal → Settings → Linked devices → **+** → scan. Provisioning links
+   expire in a couple of minutes, so generate and scan in one sitting.
+
+4. Confirm — this must list the number, not `[]`:
+
+       docker exec signal-api curl -s http://localhost:8080/v1/accounts
+
+5. Send a test message:
+
+       docker exec signal-api curl -s -X POST -H "Content-Type: application/json" \
+         -d '{"message":"test","number":"+34…","recipients":["+34…"]}' \
+         http://localhost:8080/v2/send
+
+Alerts are sent **from your own number**, so sending to yourself lands in Note
+to Self. If the device is ever unlinked from the phone, alerts stop silently —
+which is why this service is itself monitored.
+
+### If the phone says "network error"
+
+The phone is not the problem. `chat.signal.org` resolves to AWS Global
+Accelerator **dualstack** addresses with the AAAA records first, this container
+has no IPv6 address at all, and this host's IPv6 path is broken — the same edge
+that returned a bogus 404 for the Go tarball. signal-cli reaches for an
+unreachable IPv6 address and dies with `Link request error: Connection closed!`,
+while the phone can only report a failed handshake.
+
+That is what `gai.conf` (mounted at `/etc/gai.conf`) fixes. If linking starts
+failing again, check it is still mounted and that `getent ahosts chat.signal.org`
+returns an IPv4 address first.
+
+## Backups
+
+Deliberately **not** backed up. The data directory holds Signal private keys,
+and the recovery path is to link again from the phone — which takes a minute and
+does not depend on any stored artefact. Backing it up would copy a credential
+off the host to buy nothing.
+
+## Verifying
+
+```bash
+docker ps --filter name=signal-api
+docker exec signal-api curl -fsS http://localhost:8080/v1/health
+docker exec signal-api curl -fsS http://localhost:8080/v1/accounts
+docker logs signal-api --tail 50
+```