personal_infra/ansible/roles/signal_api
counterweight 3a9e1d5851
alerting: Signal via signal-cli-rest-api, and faster failure detection
Three changes: detection windows tightened, a Signal transport deployed, and
alerts attached to all 84 non-transport endpoints.

── Failing faster ──────────────────────────────────────────────────────────
The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks
"did anything arrive in the last interval", so real detection is 1-2x the
window. And a window can only ever be as tight as the push frequency - which is
why two checks moved rather than just having their numbers changed.

  liveness, cpu, ups, service-health, probes   16m -> 11m
  disk, zfs                       daily/30h -> 6-hourly/7h
  backup store + pull job         daily/30h -> 6-hourly/7h
  DNS records                           24h -> 6h
  backup dump                             30h -> 26h

backup-dump stays slow because the dump genuinely is daily. The store-side check
catches the same fault within 6h by reading the source's dump timestamp out of
the artefact filename, so 26h is a backstop rather than the primary signal.

── Thresholds differ by check type, deliberately ───────────────────────────
`failure-threshold` counts consecutive failures, but "consecutive" is a
different amount of wall-clock time per check: a push endpoint produces one
failure per heartbeat window, a pulled one per interval. The default of 3 would
mean 33 minutes on an 11m heartbeat and over a day on a 7h one.

More importantly the heartbeat window ALREADY encodes the tolerance - an 11m
window on a 5-minute push is exactly "one missed push forgiven" - so stacking a
threshold of 3 triples a tolerance that was already chosen. Hence:

  push/heartbeat endpoints   failure-threshold 1
  pulled, 5m (public)        failure-threshold 3   (= 15 minutes)
  pulled, 6h/24h (dns, domain) failure-threshold 1

── The Signal transport ────────────────────────────────────────────────────
roles/signal_api runs signal-cli-rest-api on the observability host, pinned by
digest, MODE=native.

It publishes NO PORTS. The API has no authentication of any kind - anything that
reaches it can send messages as you and read your Signal. Gatus talks to it over
a shared docker network by service name, which is also WHY the network exists:
Gatus runs in a container, so the host's loopback is unreachable from it and a
port published on 127.0.0.1 would not have worked.

MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs
Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary
per request, and alerts are rare enough that startup cost per alert is the right
trade.

Monitored - Gatus polls /v1/health over the same network path the alerts take,
so it proves the delivery route rather than mere container liveness. Deliberately
NOT backed up: the data directory holds Signal private keys and the recovery
path is to link the device again from the phone.

Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert.
If either is down, Signal is precisely what cannot deliver the alert.

── Four traps hit while linking, all now in the role README ────────────────
  * /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while
    the binary itself emits a perfectly good URI. Not worked around by switching
    MODE, which would put a JVM in the path of every alert permanently.
  * The data dir must be owned by uid 1000, not root. A root-owned 0700 dir
    cannot be traversed by the container user, so linking silently never
    completes and /v1/accounts returns "Failed to read local accounts list".
  * `docker exec` runs as ROOT while the service runs as uid 1000, so without
    --config the account is written to /root/... on the container's ephemeral
    layer. It reports success and is destroyed on the next recreate.
  * The phone reporting "network error" was IPv6: chat.signal.org resolves to
    dualstack AAAA records first, the container has no IPv6 address, and this
    host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed
    with a mounted gai.conf that prefers IPv4.

Verified: provider loads (configuredProviders=[signal]), a test message was
delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 21:40:37 +02:00
..
defaults alerting: Signal via signal-cli-rest-api, and faster failure detection 2026-09-14 21:40:37 +02:00
tasks alerting: Signal via signal-cli-rest-api, and faster failure detection 2026-09-14 21:40:37 +02:00
templates alerting: Signal via signal-cli-rest-api, and faster failure detection 2026-09-14 21:40:37 +02:00
README.md alerting: Signal via signal-cli-rest-api, and faster failure detection 2026-09-14 21:40:37 +02:00

signal_api

Runs signal-cli-rest-api on the observability host. Gatus uses it to deliver alerts over Signal.

Gatus does not speak Signal — it POSTs JSON to this service, which holds the Signal identity and does the protocol work.

It is never published, and that is not optional

This API has no authentication of any kind. No key, no token, no basic auth. Anything that can reach the port can send messages as your identity and read your Signal. So the compose file publishes no ports at all and there is no Caddy vhost.

Gatus reaches it over a shared docker network (monitoring) by service name: http://signal-api:8080. That is also why a shared network is needed rather than a published port — Gatus runs in a container, so 127.0.0.1 for Gatus is the Gatus container, not the host.

The network is created by an explicit Ansible task in both this role and gatus, so neither stack has to be deployed before the other.

MODE, and why native

Upstream offers normal, native, json-rpc and json-rpc-native. The json-rpc modes keep a resident JVM daemon and upstream describes them as "increased memory".

This VPS has 464 MB of RAM, already running Gatus and Caddy. A resident JVM is not affordable. native runs a precompiled GraalVM binary per request — no daemon, no resident cost — and alerts are rare enough that paying startup cost per alert is the right trade.

Linking the device — a one-time manual step

Ansible cannot scan a QR code, so this is manual. Do not use /v1/qrcodelink — it is broken in native mode.

The trap

GET /v1/qrcodelink?device_name=... returns:

{"error":"Couldn't create QR code: no data to encode"}

The linking itself is fine: running the binary directly inside the container emits a perfectly good provisioning URI.

$ docker exec signal-api signal-cli-native link -n gatus
sgnl://linkdevice?uuid=...&pub_key=...

It is the REST wrapper that fails to capture that output in native mode.

Do not "fix" this by switching MODE to normal or json-rpc. That puts a JVM in the path of every alert on a 464 MB host, permanently degrading the running system to work around a step performed once. Generate the QR yourself instead.

The procedure

docker exec runs as root, but the service runs as uid 1000. Without --config, signal-cli writes the linked account to /root/.local/share/signal-cli — the container's ephemeral layer, NOT the mounted volume. It looks like it worked (Associated with: +34…), /v1/accounts keeps returning [], and the account is destroyed on the next docker compose up. Always pass --config.

  1. Start the link and capture the URI. It must keep running while you scan:

    docker exec signal-api sh -c "rm -f /tmp/link.uri; \
      nohup signal-cli-native --config /home/.local/share/signal-cli \
        link -n gatus > /tmp/link.uri 2>/tmp/link.log & echo started"
    sleep 10
    docker exec signal-api cat /tmp/link.uri
    

    Do not add setsid, and do not background docker exec itself from the host — the first stops the URI appearing, the second is killed when the Ansible task returns. The output is block-buffered because stdout is a file, so the URI appears only after several seconds; stdbuf does not help, as the buffering is GraalVM's, not libc's.

  2. Render the QR on your own machine and scan it:

    qrencode -o /tmp/qr.png -s 12 -m 4 "sgnl://linkdevice?uuid=...&pub_key=..."
    
  3. Phone: Signal → Settings → Linked devices → + → scan. Provisioning links expire in a couple of minutes, so generate and scan in one sitting.

  4. Confirm — this must list the number, not []:

    docker exec signal-api curl -s http://localhost:8080/v1/accounts
    
  5. Send a test message:

    docker exec signal-api curl -s -X POST -H "Content-Type: application/json" \
      -d '{"message":"test","number":"+34…","recipients":["+34…"]}' \
      http://localhost:8080/v2/send
    

Alerts are sent from your own number, so sending to yourself lands in Note to Self. If the device is ever unlinked from the phone, alerts stop silently — which is why this service is itself monitored.

If the phone says "network error"

The phone is not the problem. chat.signal.org resolves to AWS Global Accelerator dualstack addresses with the AAAA records first, this container has no IPv6 address at all, and this host's IPv6 path is broken — the same edge that returned a bogus 404 for the Go tarball. signal-cli reaches for an unreachable IPv6 address and dies with Link request error: Connection closed!, while the phone can only report a failed handshake.

That is what gai.conf (mounted at /etc/gai.conf) fixes. If linking starts failing again, check it is still mounted and that getent ahosts chat.signal.org returns an IPv4 address first.

Backups

Deliberately not backed up. The data directory holds Signal private keys, and the recovery path is to link again from the phone — which takes a minute and does not depend on any stored artefact. Backing it up would copy a credential off the host to buy nothing.

Verifying

docker ps --filter name=signal-api
docker exec signal-api curl -fsS http://localhost:8080/v1/health
docker exec signal-api curl -fsS http://localhost:8080/v1/accounts
docker logs signal-api --tail 50