alerting: Signal via signal-cli-rest-api, and faster failure detection

Three changes: detection windows tightened, a Signal transport deployed, and
alerts attached to all 84 non-transport endpoints.

── Failing faster ──────────────────────────────────────────────────────────
The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks
"did anything arrive in the last interval", so real detection is 1-2x the
window. And a window can only ever be as tight as the push frequency - which is
why two checks moved rather than just having their numbers changed.

  liveness, cpu, ups, service-health, probes   16m -> 11m
  disk, zfs                       daily/30h -> 6-hourly/7h
  backup store + pull job         daily/30h -> 6-hourly/7h
  DNS records                           24h -> 6h
  backup dump                             30h -> 26h

backup-dump stays slow because the dump genuinely is daily. The store-side check
catches the same fault within 6h by reading the source's dump timestamp out of
the artefact filename, so 26h is a backstop rather than the primary signal.

── Thresholds differ by check type, deliberately ───────────────────────────
`failure-threshold` counts consecutive failures, but "consecutive" is a
different amount of wall-clock time per check: a push endpoint produces one
failure per heartbeat window, a pulled one per interval. The default of 3 would
mean 33 minutes on an 11m heartbeat and over a day on a 7h one.

More importantly the heartbeat window ALREADY encodes the tolerance - an 11m
window on a 5-minute push is exactly "one missed push forgiven" - so stacking a
threshold of 3 triples a tolerance that was already chosen. Hence:

  push/heartbeat endpoints   failure-threshold 1
  pulled, 5m (public)        failure-threshold 3   (= 15 minutes)
  pulled, 6h/24h (dns, domain) failure-threshold 1

── The Signal transport ────────────────────────────────────────────────────
roles/signal_api runs signal-cli-rest-api on the observability host, pinned by
digest, MODE=native.

It publishes NO PORTS. The API has no authentication of any kind - anything that
reaches it can send messages as you and read your Signal. Gatus talks to it over
a shared docker network by service name, which is also WHY the network exists:
Gatus runs in a container, so the host's loopback is unreachable from it and a
port published on 127.0.0.1 would not have worked.

MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs
Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary
per request, and alerts are rare enough that startup cost per alert is the right
trade.

Monitored - Gatus polls /v1/health over the same network path the alerts take,
so it proves the delivery route rather than mere container liveness. Deliberately
NOT backed up: the data directory holds Signal private keys and the recovery
path is to link the device again from the phone.

Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert.
If either is down, Signal is precisely what cannot deliver the alert.

── Four traps hit while linking, all now in the role README ────────────────
  * /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while
    the binary itself emits a perfectly good URI. Not worked around by switching
    MODE, which would put a JVM in the path of every alert permanently.
  * The data dir must be owned by uid 1000, not root. A root-owned 0700 dir
    cannot be traversed by the container user, so linking silently never
    completes and /v1/accounts returns "Failed to read local accounts list".
  * `docker exec` runs as ROOT while the service runs as uid 1000, so without
    --config the account is written to /root/... on the container's ephemeral
    layer. It reports success and is destroyed on the next recreate.
  * The phone reporting "network error" was IPv6: chat.signal.org resolves to
    dualstack AAAA records first, the container has no IPv6 address, and this
    host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed
    with a mounted gai.conf that prefers IPv4.

Verified: provider loads (configuredProviders=[signal]), a test message was
delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/backup_store/defaults/main.yml

@@ -17,9 +17,12 @@ backup_store_sources: []
 backup_store_check_push_base: ""
 backup_store_check_push_token: ""
 
-# Runs after the 04:00 pull. Late enough that a slow pull has finished, early
-# enough that a failure is visible before the working day.
-backup_store_check_on_calendar: "*-*-* 05:30:00"
+# Every six hours, offset past the 04:00 pull so the first run of the day sees a
+# finished pull. The BACKUPS are daily, but this check is not - it reads the
+# source's dump timestamp out of the artefact filename, so running it more often
+# catches "the source stopped dumping" within hours rather than a day, and lets
+# the Gatus heartbeat be 7h instead of 30h.
+backup_store_check_on_calendar: "*-*-* 05:30:00,11:30:00,17:30:00,23:30:00"
 
 # An artefact older than this is stale. Sources dump daily at 02:00-02:30 and the
 # pull is at 04:00, so 26h tolerates exactly one missed night before alarming.

ansible/roles/gatus/defaults/main.yml

@@ -100,3 +100,9 @@ gatus_self_check: true
 # container would get free only if it ran as root; it does not. Set false if
 # you never use icmp:// checks and want the capability dropped entirely.
 gatus_allow_icmp: true
+
+# ── Shared network ───────────────────────────────────────────────────────────
+# Gatus runs in a container, so the HOST's loopback is not reachable from it.
+# Anything Gatus must talk to locally - the Signal API that sends its alerts -
+# has to be on a shared docker network and addressed by service name.
+gatus_network: monitoring

ansible/roles/gatus/tasks/configure.yml

@@ -5,6 +5,16 @@
   changed_when: false
   failed_when: gatus_docker_check.rc != 0
 
+# Created explicitly rather than by either compose file, so neither the gatus
+# stack nor the signal-api stack has to be deployed before the other.
+- name: Ensure the shared monitoring network exists
+  ansible.builtin.command: "docker network create {{ gatus_network }}"
+  register: gatus_net
+  changed_when: "'already exists' not in gatus_net.stderr"
+  failed_when:
+    - gatus_net.rc != 0
+    - "'already exists' not in gatus_net.stderr"
+
 - name: Create the gatus directories
   ansible.builtin.file:
     path: "{{ item.path }}"

ansible/roles/gatus/templates/docker-compose.yml.j2

@@ -37,8 +37,17 @@ services:
       - NET_RAW
 {% endif %}
 
+    networks:
+      # Shared with signal-api, so alerts can be delivered by service name.
+      # 127.0.0.1 inside this container is the container, not the host.
+      - {{ gatus_network }}
+
     logging:
       driver: json-file
       options:
         max-size: "10m"
         max-file: "3"
+
+networks:
+  {{ gatus_network }}:
+    external: true

ansible/roles/gatus_endpoint/defaults/main.yml

@@ -31,3 +31,14 @@ gatus_endpoint_external: []
 gatus_config_dir: /opt/gatus/config
 gatus_endpoints_dir: "{{ gatus_config_dir }}/endpoints"
 gatus_gid: 10001
+
+# Alerts attached to every endpoint in this file that does not specify its own.
+#
+# Gatus's provider-level `default-alert` only supplies DEFAULTS - an endpoint
+# still has to opt in with `alerts: - type: signal` or it alerts on nothing at
+# all. With ~90 endpoints that cannot be written by hand, so it is applied here.
+#
+# failure-threshold is set by the CALLER, because the right value depends on the
+# check's cadence and there is no single correct default. See the note in
+# infra/400_host_monitoring.yml.
+gatus_endpoint_default_alerts: []

ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2

@@ -14,9 +14,10 @@ external-endpoints:
     heartbeat:
       interval: {{ e.heartbeat }}
 {% endif %}
-{% if e.alerts | default([]) %}
+{% set _alerts = e.alerts | default(gatus_endpoint_default_alerts) %}
+{% if _alerts %}
     alerts:
-{{ e.alerts | to_nice_yaml(indent=2) | indent(6, true) }}
+{{ _alerts | to_nice_yaml(indent=2) | indent(6, true) }}
 {% endif %}
 {% endfor %}
 {% endif %}
@@ -44,9 +45,10 @@ endpoints:
 {% for c in e.conditions %}
       - "{{ c }}"
 {% endfor %}
-{% if e.alerts | default([]) %}
+{% set _alerts = e.alerts | default(gatus_endpoint_default_alerts) %}
+{% if _alerts %}
     alerts:
-{{ e.alerts | to_nice_yaml(indent=2) | indent(6, true) }}
+{{ _alerts | to_nice_yaml(indent=2) | indent(6, true) }}
 {% endif %}
 {% endfor %}
 {% endif %}

ansible/roles/signal_api/README.md

@@ -0,0 +1,133 @@
+# signal_api
+
+Runs [signal-cli-rest-api](https://github.com/bbernhard/signal-cli-rest-api) on
+the `observability` host. Gatus uses it to deliver alerts over Signal.
+
+Gatus does not speak Signal — it POSTs JSON to this service, which holds the
+Signal identity and does the protocol work.
+
+## It is never published, and that is not optional
+
+**This API has no authentication of any kind.** No key, no token, no basic auth.
+Anything that can reach the port can send messages as your identity and read
+your Signal. So the compose file publishes **no ports at all** and there is no
+Caddy vhost.
+
+Gatus reaches it over a shared docker network (`monitoring`) by service name:
+`http://signal-api:8080`. That is also *why* a shared network is needed rather
+than a published port — Gatus runs in a container, so `127.0.0.1` for Gatus is
+the Gatus container, not the host.
+
+The network is created by an explicit Ansible task in both this role and
+`gatus`, so neither stack has to be deployed before the other.
+
+## MODE, and why `native`
+
+Upstream offers `normal`, `native`, `json-rpc` and `json-rpc-native`. The
+json-rpc modes keep a resident JVM daemon and upstream describes them as
+"increased memory".
+
+**This VPS has 464 MB of RAM**, already running Gatus and Caddy. A resident JVM
+is not affordable. `native` runs a precompiled GraalVM binary per request — no
+daemon, no resident cost — and alerts are rare enough that paying startup cost
+per alert is the right trade.
+
+## Linking the device — a one-time manual step
+
+Ansible cannot scan a QR code, so this is manual. **Do not use
+`/v1/qrcodelink`** — it is broken in `native` mode.
+
+### The trap
+
+`GET /v1/qrcodelink?device_name=...` returns:
+
+```json
+{"error":"Couldn't create QR code: no data to encode"}
+```
+
+The linking itself is fine: running the binary directly inside the container
+emits a perfectly good provisioning URI.
+
+```
+$ docker exec signal-api signal-cli-native link -n gatus
+sgnl://linkdevice?uuid=...&pub_key=...
+```
+
+It is the REST wrapper that fails to capture that output in `native` mode.
+
+**Do not "fix" this by switching MODE to `normal` or `json-rpc`.** That puts a
+JVM in the path of *every alert* on a 464 MB host, permanently degrading the
+running system to work around a step performed once. Generate the QR yourself
+instead.
+
+### The procedure
+
+**`docker exec` runs as root, but the service runs as uid 1000.** Without
+`--config`, signal-cli writes the linked account to `/root/.local/share/signal-cli`
+— the container's ephemeral layer, NOT the mounted volume. It looks like it
+worked (`Associated with: +34…`), `/v1/accounts` keeps returning `[]`, and the
+account is destroyed on the next `docker compose up`. Always pass `--config`.
+
+1. Start the link and capture the URI. It must keep running while you scan:
+
+       docker exec signal-api sh -c "rm -f /tmp/link.uri; \
+         nohup signal-cli-native --config /home/.local/share/signal-cli \
+           link -n gatus > /tmp/link.uri 2>/tmp/link.log & echo started"
+       sleep 10
+       docker exec signal-api cat /tmp/link.uri
+
+   Do **not** add `setsid`, and do **not** background `docker exec` itself from
+   the host — the first stops the URI appearing, the second is killed when the
+   Ansible task returns. The output is block-buffered because stdout is a file,
+   so the URI appears only after several seconds; `stdbuf` does not help, as the
+   buffering is GraalVM's, not libc's.
+
+2. Render the QR on your own machine and scan it:
+
+       qrencode -o /tmp/qr.png -s 12 -m 4 "sgnl://linkdevice?uuid=...&pub_key=..."
+
+3. Phone: Signal → Settings → Linked devices → **+** → scan. Provisioning links
+   expire in a couple of minutes, so generate and scan in one sitting.
+
+4. Confirm — this must list the number, not `[]`:
+
+       docker exec signal-api curl -s http://localhost:8080/v1/accounts
+
+5. Send a test message:
+
+       docker exec signal-api curl -s -X POST -H "Content-Type: application/json" \
+         -d '{"message":"test","number":"+34…","recipients":["+34…"]}' \
+         http://localhost:8080/v2/send
+
+Alerts are sent **from your own number**, so sending to yourself lands in Note
+to Self. If the device is ever unlinked from the phone, alerts stop silently —
+which is why this service is itself monitored.
+
+### If the phone says "network error"
+
+The phone is not the problem. `chat.signal.org` resolves to AWS Global
+Accelerator **dualstack** addresses with the AAAA records first, this container
+has no IPv6 address at all, and this host's IPv6 path is broken — the same edge
+that returned a bogus 404 for the Go tarball. signal-cli reaches for an
+unreachable IPv6 address and dies with `Link request error: Connection closed!`,
+while the phone can only report a failed handshake.
+
+That is what `gai.conf` (mounted at `/etc/gai.conf`) fixes. If linking starts
+failing again, check it is still mounted and that `getent ahosts chat.signal.org`
+returns an IPv4 address first.
+
+## Backups
+
+Deliberately **not** backed up. The data directory holds Signal private keys,
+and the recovery path is to link again from the phone — which takes a minute and
+does not depend on any stored artefact. Backing it up would copy a credential
+off the host to buy nothing.
+
+## Verifying
+
+```bash
+docker ps --filter name=signal-api
+docker exec signal-api curl -fsS http://localhost:8080/v1/health
+docker exec signal-api curl -fsS http://localhost:8080/v1/accounts
+docker logs signal-api --tail 50
+```

ansible/roles/signal_api/defaults/main.yml

@@ -0,0 +1,39 @@
+---
+# signal-cli-rest-api: the transport Gatus uses to send Signal messages.
+#
+# Gatus does not speak Signal. It POSTs JSON to this service, which holds the
+# actual Signal identity and does the protocol work.
+
+# Pinned by digest for the same reason as Gatus: a tag is mutable.
+# Upstream publishes no versioned tags worth pinning to, so this pins the
+# DIGEST that `latest` resolved to when this was reviewed. `latest` is a moving
+# target; a digest is a content address, and `docker compose pull` either
+# fetches exactly this image or fails.
+signal_api_image_digest: "sha256:2399d449123cdad56c4d859277e3b9127e1a00c4d2ab4601c239882609286cf8"
+signal_api_image: "bbernhard/signal-cli-rest-api@{{ signal_api_image_digest }}"
+
+signal_api_dir: /opt/signal-api
+signal_api_data_dir: "{{ signal_api_dir }}/data"
+
+# MODE matters on this host. Upstream offers normal / native / json-rpc /
+# json-rpc-native. json-rpc keeps a resident JVM daemon and upstream describes it
+# as "increased memory" - this VPS has 464MB total and already runs Gatus and
+# Caddy, so a resident JVM is not affordable. `native` runs a precompiled
+# GraalVM binary per request: no daemon, no resident cost, and alerts are rare
+# enough that paying startup per alert is the right trade.
+signal_api_mode: native
+
+# Port INSIDE the shared docker network. Never published to the host: this API
+# has NO AUTHENTICATION of any kind. Anyone who can reach it can send messages
+# as you and read your Signal.
+signal_api_port: 8080
+
+# Both this and Gatus join this network so Gatus can reach the API by service
+# name. Gatus runs in a container, so the host's loopback is NOT reachable from
+# it - this is why a shared network is required rather than a published port.
+signal_api_network: monitoring
+signal_api_service_name: signal-api
+
+# The uid the upstream image drops to (`setpriv --reuid=1000`). The data
+# directory must be owned by it or signal-cli cannot write the account.
+signal_api_uid: 1000

ansible/roles/signal_api/tasks/main.yml

@@ -0,0 +1,93 @@
+---
+- name: Assert Docker is available
+  ansible.builtin.command: docker --version
+  register: signal_docker_check
+  changed_when: false
+
+# Created explicitly rather than by either compose file, so neither stack has to
+# be deployed before the other and neither owns it.
+- name: Ensure the shared monitoring network exists
+  ansible.builtin.command: "docker network create {{ signal_api_network }}"
+  register: signal_net
+  changed_when: "'already exists' not in signal_net.stderr"
+  failed_when:
+    - signal_net.rc != 0
+    - "'already exists' not in signal_net.stderr"
+
+- name: Create the signal-api directory
+  ansible.builtin.file:
+    path: "{{ signal_api_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+# Owned by the container's uid, NOT root.
+#
+# The image drops to uid 1000 (`setpriv --reuid=1000`), and a root-owned 0700
+# directory cannot be traversed by uid 1000 - signal-cli then fails to write the
+# account and linking silently never completes, leaving a 39-byte accounts.json
+# with no accounts and the API returning "Failed to read local accounts list".
+#
+# 0700 on uid 1000 is still private: only that uid and root can read the Signal
+# private keys, which is the property actually wanted.
+- name: Create the signal-api data directory owned by the container user
+  ansible.builtin.file:
+    path: "{{ signal_api_data_dir }}"
+    state: directory
+    owner: "{{ signal_api_uid }}"
+    group: "{{ signal_api_uid }}"
+    mode: "0700"
+
+- name: Write the IPv4-preference resolver config
+  ansible.builtin.template:
+    src: gai.conf.j2
+    dest: "{{ signal_api_dir }}/gai.conf"
+    owner: root
+    group: root
+    mode: "0644"
+
+- name: Write the docker compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ signal_api_dir }}/docker-compose.yml"
+    owner: root
+    group: root
+    mode: "0644"
+
+- name: Pull the pinned signal-api image
+  ansible.builtin.command:
+    cmd: docker compose pull
+    chdir: "{{ signal_api_dir }}"
+  register: signal_pull
+  changed_when: "'Downloaded newer image' in signal_pull.stderr or 'Pull complete' in signal_pull.stderr"
+
+- name: Start signal-api
+  ansible.builtin.command:
+    cmd: docker compose up -d --remove-orphans
+    chdir: "{{ signal_api_dir }}"
+  register: signal_up
+  changed_when: "'Started' in signal_up.stderr or 'Created' in signal_up.stderr or 'Recreated' in signal_up.stderr"
+
+- name: Wait for the API to answer
+  ansible.builtin.command:
+    cmd: "docker exec {{ signal_api_service_name }} curl -fsS http://localhost:{{ signal_api_port }}/v1/health"
+  register: signal_health
+  until: signal_health.rc == 0
+  retries: 12
+  delay: 5
+  changed_when: false
+
+- name: Report whether an account is linked yet
+  ansible.builtin.command:
+    cmd: "docker exec {{ signal_api_service_name }} curl -fsS http://localhost:{{ signal_api_port }}/v1/accounts"
+  register: signal_accounts
+  changed_when: false
+  failed_when: false
+
+- name: Show the linking status
+  ansible.builtin.debug:
+    msg: >-
+      {{ 'Linked account(s): ' ~ signal_accounts.stdout
+         if (signal_accounts.stdout | default('[]') | trim) not in ['[]', '', 'null']
+         else 'NO ACCOUNT LINKED YET - this is a one-time manual step, see the role README.' }}

ansible/roles/signal_api/templates/docker-compose.yml.j2

@@ -0,0 +1,47 @@
+# Managed by Ansible (roles/signal_api)
+services:
+  {{ signal_api_service_name }}:
+    image: {{ signal_api_image }}
+    container_name: {{ signal_api_service_name }}
+    restart: unless-stopped
+
+    environment:
+      MODE: "{{ signal_api_mode }}"
+
+    volumes:
+      # Prefer IPv4. See gai.conf.j2 - without this, signal-cli reaches for
+      # chat.signal.org's IPv6 address, which is unreachable from here, and
+      # linking fails with an opaque "network error" on the phone.
+      - {{ signal_api_dir }}/gai.conf:/etc/gai.conf:ro
+      # Holds the Signal identity: the linked-device keys and registration
+      # state. Lose this and the device must be linked again by scanning a new
+      # QR code from the phone. It is also the most sensitive thing on this
+      # host - anyone with these keys can send and read Signal as you.
+      - {{ signal_api_data_dir }}:/home/.local/share/signal-cli
+
+    networks:
+      - {{ signal_api_network }}
+
+    # NO PORTS. Deliberately.
+    #
+    # This API has no authentication whatsoever - no key, no token, nothing.
+    # Publishing it, even on 127.0.0.1, would expose "send a Signal message as
+    # this identity" to anything that can reach the host. Gatus talks to it over
+    # the shared docker network by service name instead, which is why no port is
+    # published and why there is no Caddy vhost.
+
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://localhost:8080/v1/health"]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+networks:
+  {{ signal_api_network }}:
+    external: true

ansible/roles/signal_api/templates/gai.conf.j2

@@ -0,0 +1,17 @@
+# Managed by Ansible (roles/signal_api)
+#
+# Prefer IPv4 over IPv6 in getaddrinfo.
+#
+# chat.signal.org resolves to AWS Global Accelerator dualstack addresses, and
+# DNS returns the AAAA records first. This container has NO IPv6 address at all,
+# and this host's IPv6 path is unreliable anyway - the same edge that made
+# Google's IPv6 endpoint return a confident 404 for the Go tarball.
+#
+# signal-cli would connect to the AAAA address, fail, and report
+#   Link request error: Connection closed!
+# while the phone showed a bare "network error" - a failure with no obvious
+# cause on either end.
+#
+# This line flips the precedence so IPv4-mapped addresses sort first, which is
+# the standard glibc fix. It does NOT disable IPv6; it only changes the order.
+precedence ::ffff:0:0/96  100