counterweight
fa9f7d10cd
First step of replacing Uptime Kuma and ntfy. Gatus runs on the new
observability VPS, fronted by Caddy at status.contrapeso.xyz.
Deployed as the upstream container image, not built from source. The role did
build from source first - their Dockerfile is a bare `CGO_ENABLED=0 go build`,
the Vue dashboard is compiled in via `//go:embed static` in web/static.go, and
CGO can stay off because the sqlite driver is pure-Go modernc.org/sqlite - but
that produces a binary upstream never ran, and it meant compiling the AWS SDK
and gRPC on the smallest box in the estate. That load was heavy enough that
unrelated Ansible tasks timed out while it ran. The cost of the container is a
daemon on the machine whose job is to notice when everything else breaks; that
trade is made deliberately and is written down in the role README.
Pinned by DIGEST, not tag. A tag is mutable - v5.36.0 can be repushed - so
pinning it alone is a weaker promise than it looks:
gatus_image: "ghcr.io/twin/gatus@sha256:c5f210d0..."
`docker compose pull` now either fetches exactly the reviewed image or fails.
gatus_version is kept beside it only so a human can read the release; the two
move together.
The image is FROM scratch, so it has no /etc/passwd and its default user is
root. The container runs as 10001:10001 with the host data dir owned to match,
plus read_only, cap_drop ALL, and no-new-privileges. NET_RAW is added back only
when gatus_allow_icmp, so the capability for icmp:// checks is a visible grant
rather than something inherited from running as root.
Config is a DIRECTORY, not a file. Gatus merges every *.yaml under
GATUS_CONFIG_PATH - maps deep-merge, lists append - so the role owns
00-base.yaml (web, storage, ui, alerting, security) and each service will drop
its own file into endpoints/, the same shape as caddy_site. A primitive defined
twice is ambiguous and upstream refuses it, so anything that is not a list lives
in the base file and nowhere else.
Two bugs the deploy caught:
* Gatus panics on a config with no endpoints ("configuration should contain at
least one endpoint or suite"), so "install now, add endpoints later" is not
a valid state. The role ships endpoints/00-self.yaml checking its own
/health. Less circular than it looks: it proves the directory merged, the
listener serves, and storage accepted a write.
* web.address was carried over from the systemd design as 127.0.0.1. Inside a
container that is the CONTAINER's loopback, which docker-proxy cannot reach
- gatus came up healthy, self-check passing, while every connection to the
published port was refused. It now always binds 0.0.0.0 inside the
container; the isolation comes from publishing to 127.0.0.1 on the host.
Auth is done at the edge, NOT with Gatus's own security.basic. Reading
api/api.go, that middleware protects exactly four routes - the statuses
endpoints. Everything else is registered on the unprotected router, including
/api/v1/config, every badge, and /api/v1/endpoints/:key/uptimes/:duration and
.../response-times/:duration/history, which return real data to anyone who can
guess a key ("<group>_<name>"). Verified against the live instance: all seven
routes returned 200 unauthenticated, and /uptimes/24h returned "1.000000".
So the vhost uses caddy_site_body with a path carve-out rather than
caddy_site_basic_auth, which has no way to exempt a path. The external-endpoint
push API must NOT sit behind basic auth: it authenticates with
`Authorization: Bearer <token>`, and basic auth wants the same header. It is not
unauthenticated - the handler 401s on a missing prefix, an empty token, or a
token that does not match that endpoint's own.
Verified end to end. All seven previously-open routes now 401. The push path
distinguishes cleanly: POST with no auth gets Gatus's own "invalid Authorization
header" with NO WWW-Authenticate; POST with a bogus Bearer gets 404 (key looked
up, no external endpoints yet); GET on the same path gets Caddy's 401 with
WWW-Authenticate: Basic, so the exemption is scoped to POST alone. The
self-check still passes because it polls localhost inside the container and
never traverses Caddy.
The host itself was rebuilt from scratch: 01 (ok=9 changed=8), 02 (ok=12
changed=6), 910_docker (--limit, since that playbook still wrongly claims all of
`managed` needs Docker), caddy (ok=13 changed=8), gatus (ok=18 changed=2).
Not done here: gatus_alerting is still {} - valid, and every condition is
evaluated and recorded, there is just nowhere to shout until a provider is
chosen to replace ntfy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
83 lines
3.6 KiB
YAML
83 lines
3.6 KiB
YAML
---
|
|
# Gatus: health checks, status page and alerting for the whole estate.
|
|
#
|
|
# Built from source and run under systemd - upstream publishes no binaries, and
|
|
# their Dockerfile shows the runtime needs nothing but the static binary and a
|
|
# CA bundle. See roles/gatus/README.md.
|
|
- name: Deploy Gatus on the observability host
|
|
hosts: observability
|
|
become: yes
|
|
roles:
|
|
- gatus
|
|
|
|
# The dashboard is bound to loopback; Caddy publishes it.
|
|
#
|
|
# Auth is done HERE, at the edge, and not with Gatus's own `security.basic`.
|
|
# Gatus's security middleware protects exactly four routes (api/api.go):
|
|
#
|
|
# /api/v1/endpoints/statuses
|
|
# /api/v1/endpoints/:key/statuses
|
|
# /api/v1/suites/statuses
|
|
# /api/v1/suites/:key/statuses
|
|
#
|
|
# Everything else is registered on the UNPROTECTED router, including
|
|
# /api/v1/config, every badge, and - the part that matters -
|
|
# /api/v1/endpoints/:key/uptimes/:duration and .../response-times/:duration/history,
|
|
# which return real per-endpoint data to anyone who can guess a key. Keys are
|
|
# just "<group>_<name>". So Gatus's own auth makes the dashboard render empty
|
|
# while leaving the data readable, which is worse than it looks.
|
|
#
|
|
# The one route that must NOT sit behind basic auth is the external-endpoint
|
|
# push API. It authenticates with `Authorization: Bearer <token>`, and basic
|
|
# auth wants `Authorization: Basic <...>` - same header, two schemes, and the
|
|
# push clients lose. It is not actually unauthenticated: the handler 401s on a
|
|
# missing prefix, an empty token, or a token that does not match that endpoint's
|
|
# own. Upstream's comment on the route says exactly that.
|
|
- name: Publish the Gatus status page through Caddy
|
|
hosts: observability
|
|
become: yes
|
|
tasks:
|
|
- name: Require the dashboard credentials to be set
|
|
ansible.builtin.assert:
|
|
that:
|
|
- gatus_dashboard_username is defined
|
|
- gatus_dashboard_username | length > 0
|
|
- gatus_dashboard_password_hash is defined
|
|
- gatus_dashboard_password_hash.startswith('$2')
|
|
fail_msg: >-
|
|
gatus_dashboard_username and gatus_dashboard_password_hash must be in
|
|
the vault. Generate the hash on the observability host, which runs
|
|
Caddy natively, so the bcrypt cost and format match what verifies it:
|
|
caddy hash-password --plaintext 'your-password'
|
|
then: ansible-vault edit group_vars/all/vault.yml
|
|
|
|
- name: Configure the Caddy vhost for Gatus
|
|
ansible.builtin.include_role:
|
|
name: caddy_site
|
|
vars:
|
|
caddy_site_name: gatus
|
|
caddy_site_domain: "{{ subdomains.gatus }}.{{ root_domain }}"
|
|
# caddy_site_body rather than caddy_site_upstream + caddy_site_basic_auth,
|
|
# because that pair applies auth to the whole site with no way to carve
|
|
# out the push path. `handle` blocks are mutually exclusive and first
|
|
# match wins, so the push API gets a route of its own.
|
|
caddy_site_body: |
|
|
@push {
|
|
path /api/v1/endpoints/*/external
|
|
method POST
|
|
}
|
|
|
|
# Push API: Bearer-authenticated by Gatus itself. No basic auth here,
|
|
# or the Authorization header collides.
|
|
handle @push {
|
|
reverse_proxy 127.0.0.1:{{ gatus_port | default(8080) }}
|
|
}
|
|
|
|
# Everything else: the dashboard, the config endpoint, the badges and
|
|
# the uptime/response-time history.
|
|
handle {
|
|
basic_auth {
|
|
{{ gatus_dashboard_username }} {{ gatus_dashboard_password_hash }}
|
|
}
|
|
reverse_proxy 127.0.0.1:{{ gatus_port | default(8080) }}
|
|
}
|