monitoring: systemd services, domain expiry, DNS correctness, public endpoints
Four more check types, 42 endpoints, taking the estate from 39 to 81.
── systemd services (infra/401) ────────────────────────────────────────────
Every unit we deploy, checked every 5 minutes with a 16-minute heartbeat.
This closes the gap that let a real bug run unnoticed earlier today: a backup
script left forgejo, lnbits, headscale and memos stopped, and NOTHING caught
it. The dumps exited 0, the artefacts were correct, the deploy said failed=0,
and liveness only proves the HOST is up - not that anything on it serves.
One endpoint PER UNIT but only ONE timer per host: the check iterates that
host's units and pushes a result for each, the way check-backups.sh reports per
source. Four units on vipy would otherwise mean four scripts, services and
timers. A single host-level red light would also say "something on vipy is
down" without saying which, which is not the question anyone has.
Keys are host-qualified because unit names collide - caddy runs on four
machines. Which units a host runs lives in host_vars/<host>/monitored_services,
because "what runs here" is a property of the machine, the same reasoning as
the cross-host ports.
nut-driver-enumerator is deliberately excluded: it is a oneshot generator that
is `enabled` but always `inactive`, so it would report down forever. Checked
live before excluding it.
── domain, DNS and public endpoints (infra/402) ────────────────────────────
The first checks that PULL rather than push, and that is the right way round:
all three are about how the outside world sees us, so they must be measured
from outside. Nothing is installed anywhere - no script, no timer, no token.
They also have no heartbeat, because a heartbeat answers "did the reporter
report"; when Gatus does the checking itself, failure is immediate.
domain 1 endpoint, 24h, [DOMAIN_EXPIRATION] > 336h (two weeks)
dns 11 endpoints, 24h, [DNS_RCODE] == NOERROR and [BODY] == the IP
public 14 endpoints, 5m, 11 HTTPS + 3 TCP
Expected A records are derived from inventory (hostvars[host].ansible_host),
not written down again. The estate's recurring bug is an address recorded in a
second place and left behind when the machine moved; asserting against
inventory means a renumbered box is one edit, not two.
Expected HTTP status was checked live per site rather than assumed. Two return
401 - the Gatus dashboard and the DATUM dashboard, both behind basic auth - and
that is what is asserted: expecting 200 there would go green precisely when the
auth broke. headscale asserts /health rather than /, which is a 404 by design.
Every HTTPS check also carries [CERTIFICATE_EXPIRATION] > 168h, which is free
on an endpoint already being polled and catches a renewal that silently stops.
Two things learned the hard way, both now in comments:
* A domain-expiry endpoint needs a URL SCHEME. Gatus derives the endpoint type
from the prefix (endpoint.Type()), so a bare "contrapeso.xyz" is UNKNOWN and
the whole config is rejected. It is https:// plus a DOMAIN_EXPIRATION
condition and no status assertion, so the registrar's parking page at the
apex is irrelevant. Upstream also enforces a 5m minimum interval for that
placeholder, because it uses a free whois service.
* That rejection proved skip-invalid-config-update was worth adding. Gatus
logged "the configuration file was updated, but it is not valid, the old
configuration will continue being used" and kept running. Without it the
reload path calls panic() and one malformed contributed file takes the
monitor down.
Verified: 15/15 service endpoints UP; 26/27 public-facing UP. The single DOWN is
public_memos, correctly - memos-box is powered off, and the condition result
reads [STATUS] (502) == 200. memos-box and arbret-staging-box were shut down
deliberately from the Proxmox UI to test the liveness endpoints; their endpoints
stay registered and will go green when the VMs return.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 09:33:57 +02:00
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# Systemd services deployed on this host, monitored every 5 minutes.
|
|
|
|
|
#
|
|
|
|
|
# The fact lives with the machine rather than in a central map, for the same
|
|
|
|
|
# reason the cross-host ports do: "what runs here" is a property of the host,
|
|
|
|
|
# and a central list is one more thing to forget to update when a service moves.
|
|
|
|
|
#
|
|
|
|
|
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
|
|
|
|
|
# supervision and would be noise.
|
|
|
|
|
monitored_services:
|
|
|
|
|
- forgejo-runner
|
forgejo-runner: give CI jobs a Docker daemon of their own
jar-lid-archive's CI needs `docker build` and `docker compose`, and job
containers had no Docker at all: config.yml carried the stock
`container.docker_host: "-"`, which is precisely "find a daemon for the
runner and do not let jobs near it".
A docker:dind sidecar now runs on forgejo-runner-box, published on
127.0.0.1 only, and jobs build against that instead of the host daemon.
Following Forgejo's own admin docs, which make the point that this is
THREE settings and not one:
forgejo-dind.service a privileged docker:dind on the host daemon
container.docker_host tcp://127.0.0.1:2376 - where the runner
CREATES job containers
runner.envs.DOCKER_HOST + tcp://forgejo-dind.docker.internal:2375 -
container.options how a job, from INSIDE dind, reaches it
The third is the one that is easy to miss and fails identically to
having done nothing: for a tcp:// docker_host the runner mounts nothing
into the job container, so the job still finds no daemon. The two
addresses differ because they are used from different places - the
runner is a process on the host; a job container is inside dind, where
127.0.0.1 is itself. `host-gateway` is expanded by the daemon creating
the container, which is dind, so it lands on dind's own bridge gateway.
Verified on the box, along the exact path a job takes - a container on a
user-defined network, with the same --add-host and DOCKER_HOST the
runner will set: the name resolves to 172.20.0.1, `docker version`
returns the dind server, and `docker build` succeeds. The host daemon
runs forgejo-dind and nothing else.
config.yml had to become a template to do any of this. It was previously
`generate-config` run once on first deploy and never touched again, so
the repo had no say in its contents. Every value was transcribed from
the live file and diffed against it key by key: with dind disabled, 30
of 30 identical. Two deliberate differences - runner.envs drops
A_TEST_ENV_NAME_1/2, upstream's placeholders, which were being injected
into every job here; and the upstream comment blob is gone. Because the
file is managed now it also gets a restart handler, the absence of which
has bitten this repo before.
Things deliberately chosen rather than defaulted into:
- the flag lives in the playbook, not the role defaults, so the role's
default stays the configuration this host has always run and turning
it off is one line. Disabling it also tears the units down rather than
just skipping them.
- dind is a hard Requires= of forgejo-runner, and its unit stays in
"activating" until the daemon answers. A dead dind is then one red
unit instead of a green runner failing every job it claims.
- the healthcheck and monitored_services learned about dind for the same
reason: runner up + dind down is the interesting failure and it was
previously invisible.
- /var/lib/docker is a named volume, or every image is re-pulled on each
container recreation; a weekly prune keeps that from growing forever.
- dind's network pool is pinned to 172.20.0.0/14. Docker's stock pool
reaches into 192.168.x, and act_runner creates a network per job, so a
job could eventually allocate this LAN's subnet and lose the route to
every box on it.
Note this runner is SHARED - arbret, arbret-biz, arbretdos and
arbret-ops build here too, far more often, and docker_host is per-runner
not per-repo. Their workflows are unchanged but their next run pulls
images into a cold cache. Job images still need a docker client of their
own: node:{20,22}-bookworm ship none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 22:32:08 +02:00
|
|
|
# The isolated Docker daemon that Actions jobs build on. Listed separately
|
|
|
|
|
# from forgejo-runner because they fail separately and the interesting case
|
|
|
|
|
# is the asymmetric one: dind down, runner up, every job failing while the
|
|
|
|
|
# runner's own unit stays green.
|
|
|
|
|
- forgejo-dind
|