From fa9f7d10cd5598db72bfbf6708023e1639baa888 Mon Sep 17 00:00:00 2001 From: counterweight Date: Sun, 13 Sep 2026 22:29:03 +0200 Subject: [PATCH] gatus: deploy on prd-monitoring, behind Caddy basic auth First step of replacing Uptime Kuma and ntfy. Gatus runs on the new observability VPS, fronted by Caddy at status.contrapeso.xyz. Deployed as the upstream container image, not built from source. The role did build from source first - their Dockerfile is a bare `CGO_ENABLED=0 go build`, the Vue dashboard is compiled in via `//go:embed static` in web/static.go, and CGO can stay off because the sqlite driver is pure-Go modernc.org/sqlite - but that produces a binary upstream never ran, and it meant compiling the AWS SDK and gRPC on the smallest box in the estate. That load was heavy enough that unrelated Ansible tasks timed out while it ran. The cost of the container is a daemon on the machine whose job is to notice when everything else breaks; that trade is made deliberately and is written down in the role README. Pinned by DIGEST, not tag. A tag is mutable - v5.36.0 can be repushed - so pinning it alone is a weaker promise than it looks: gatus_image: "ghcr.io/twin/gatus@sha256:c5f210d0..." `docker compose pull` now either fetches exactly the reviewed image or fails. gatus_version is kept beside it only so a human can read the release; the two move together. The image is FROM scratch, so it has no /etc/passwd and its default user is root. The container runs as 10001:10001 with the host data dir owned to match, plus read_only, cap_drop ALL, and no-new-privileges. NET_RAW is added back only when gatus_allow_icmp, so the capability for icmp:// checks is a visible grant rather than something inherited from running as root. Config is a DIRECTORY, not a file. Gatus merges every *.yaml under GATUS_CONFIG_PATH - maps deep-merge, lists append - so the role owns 00-base.yaml (web, storage, ui, alerting, security) and each service will drop its own file into endpoints/, the same shape as caddy_site. A primitive defined twice is ambiguous and upstream refuses it, so anything that is not a list lives in the base file and nowhere else. Two bugs the deploy caught: * Gatus panics on a config with no endpoints ("configuration should contain at least one endpoint or suite"), so "install now, add endpoints later" is not a valid state. The role ships endpoints/00-self.yaml checking its own /health. Less circular than it looks: it proves the directory merged, the listener serves, and storage accepted a write. * web.address was carried over from the systemd design as 127.0.0.1. Inside a container that is the CONTAINER's loopback, which docker-proxy cannot reach - gatus came up healthy, self-check passing, while every connection to the published port was refused. It now always binds 0.0.0.0 inside the container; the isolation comes from publishing to 127.0.0.1 on the host. Auth is done at the edge, NOT with Gatus's own security.basic. Reading api/api.go, that middleware protects exactly four routes - the statuses endpoints. Everything else is registered on the unprotected router, including /api/v1/config, every badge, and /api/v1/endpoints/:key/uptimes/:duration and .../response-times/:duration/history, which return real data to anyone who can guess a key ("_"). Verified against the live instance: all seven routes returned 200 unauthenticated, and /uptimes/24h returned "1.000000". So the vhost uses caddy_site_body with a path carve-out rather than caddy_site_basic_auth, which has no way to exempt a path. The external-endpoint push API must NOT sit behind basic auth: it authenticates with `Authorization: Bearer `, and basic auth wants the same header. It is not unauthenticated - the handler 401s on a missing prefix, an empty token, or a token that does not match that endpoint's own. Verified end to end. All seven previously-open routes now 401. The push path distinguishes cleanly: POST with no auth gets Gatus's own "invalid Authorization header" with NO WWW-Authenticate; POST with a bogus Bearer gets 404 (key looked up, no external endpoints yet); GET on the same path gets Caddy's 401 with WWW-Authenticate: Basic, so the exemption is scoped to POST alone. The self-check still passes because it polls localhost inside the container and never traverses Caddy. The host itself was rebuilt from scratch: 01 (ok=9 changed=8), 02 (ok=12 changed=6), 910_docker (--limit, since that playbook still wrongly claims all of `managed` needs Docker), caddy (ok=13 changed=8), gatus (ok=18 changed=2). Not done here: gatus_alerting is still {} - valid, and every condition is evaluated and recorded, there is just nowhere to shout until a provider is chosen to replace ntfy. Co-Authored-By: Claude Opus 5 (1M context) --- ansible/group_vars/all/main.yml | 3 +- ansible/group_vars/all/vault.yml | 393 +++++++++--------- ansible/roles/gatus/README.md | 110 +++++ ansible/roles/gatus/defaults/main.yml | 83 ++++ ansible/roles/gatus/handlers/main.yml | 5 + ansible/roles/gatus/tasks/configure.yml | 54 +++ ansible/roles/gatus/tasks/main.yml | 5 + ansible/roles/gatus/tasks/service.yml | 35 ++ ansible/roles/gatus/templates/config.yaml.j2 | 56 +++ .../gatus/templates/docker-compose.yml.j2 | 44 ++ .../gatus/templates/endpoint-self.yaml.j2 | 25 ++ .../services/gatus/deploy_gatus_playbook.yml | 83 ++++ 12 files changed, 702 insertions(+), 194 deletions(-) create mode 100644 ansible/roles/gatus/README.md create mode 100644 ansible/roles/gatus/defaults/main.yml create mode 100644 ansible/roles/gatus/handlers/main.yml create mode 100644 ansible/roles/gatus/tasks/configure.yml create mode 100644 ansible/roles/gatus/tasks/main.yml create mode 100644 ansible/roles/gatus/tasks/service.yml create mode 100644 ansible/roles/gatus/templates/config.yaml.j2 create mode 100644 ansible/roles/gatus/templates/docker-compose.yml.j2 create mode 100644 ansible/roles/gatus/templates/endpoint-self.yaml.j2 create mode 100644 ansible/services/gatus/deploy_gatus_playbook.yml diff --git a/ansible/group_vars/all/main.yml b/ansible/group_vars/all/main.yml index 1dc3541..9cf3143 100644 --- a/ansible/group_vars/all/main.yml +++ b/ansible/group_vars/all/main.yml @@ -25,7 +25,8 @@ backup_pull_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfIixKMhA9z+Nvyx6T # vars_files: - a file everyone must opt into is a file someone will forget. # ───────────────────────────────────────────────────────────────────────────── subdomains: - # Monitoring (watchtower) + # Monitoring + gatus: status ntfy: ntfy # Uptime Kuma IS still running and this subdomain DOES resolve # (164.92.239.72, HTTP 302). Only the Ansible code and the vault credentials diff --git a/ansible/group_vars/all/vault.yml b/ansible/group_vars/all/vault.yml index a619094..e8af624 100644 --- a/ansible/group_vars/all/vault.yml +++ b/ansible/group_vars/all/vault.yml @@ -1,194 +1,201 @@ $ANSIBLE_VAULT;1.1;AES256 -66646566313430366435336438316665363536356431633739373931326363363137653964333765 -6438363638393236306464663136663061386361363432390a623836393662636238626337373766 -39313636386137363334313465373637613363313634363230643032386132366533323934393933 -3033616238356165630a613765643762336161663863393163616636636163376537396638666139 -62376334383839373131646334383039396661396262633537333863613133346363303133353765 -61323462646434313539633833616162376637613436626365336265333863356534626339313661 -35376634386233653964353437373838343066323035366263663738643032646631633236663066 -34383535333961313439386462326264616535366434343032633639646631373930353038356330 -37626465653362366666316363333663343734663939393637373234303935363136366636306534 -65356362313662356332626530363563333337613661626531326138633936316137626334616162 -30336664323436653431393061343935386461383130356437646361333431313938393236363633 -39303063653936353736363332386366623565313861613838303735356530663138323935323066 -66613134653235353364366263343138666639396430306439396633393433636236363363393164 -66326330313530393064656331386233363466386336313430333139643830626535653538353331 -62643631366536653137393733666238386230383634356532663563343765306334383331333161 -35313336643266653231333034626139636133653236323235346238323335663934626133616664 -65326333323334663265303933353035353739323062323962396661666161333236666133343235 -65616562346131613136363932353632343935353139386234656138303261636230333264363230 -33333933623965343862393065313238306635636361373237626233353634373737336532626431 -30636163363835366330326163616539303031393561313538393338323166613739663039396566 -61613265653633386338663837396135356563353439336534633064363330303863333166393966 -35366337353636366336346436633734363833363133366561333835306433333138666134306462 -38626333663534386237303739646664663363303866656433313733393832316533383136316233 -66363234656664346465636562643832323266346631653262373166343036363532646533313636 -63303538656662383430623365383133313665393163333832626361386663346132313163313061 -39613732366435343930346639643161656165656538653531653036393764663962363832663861 -33333031353234646239386534623536376131306261653165393634383762383265313136326636 -33333430323030636164346433316337663137643133626639303536323735623637333031356633 -36613232313330636636633266636461616463376463366234303630613966363738633133636263 -64646532333764343861633935633737323639333237643831353339636235316533613266653865 -36333535333365333436653638663364386437613136313239376538323439613965343339656461 -39616239346435343665333831363134643762646532393366366533623461353964383263356630 -31373239666266613133323333323336363739346236656261323365363664303630663934653863 -34636534376633333331356162393339653762303837363765313633333535386164386439386266 -39303336646465366336316263343461613336303930653963613866663738333433326565643736 -61303631656132653064303439646637373734313364303135643963353064363061663835663632 -31623264383238323734373732393733343336323630366439303466316238336566623666303433 -34396633653762303730323334356166303134376535316131333266303438313562306135343233 -30303136333730626561656639623866666663646131383666613633663966636462663031626639 -65383732313561376261333534643937383165616137656232623664386232353337343563333262 -34633364323839353136616636366462666637313438396661303663316263353636363163633236 -64326265623262323735396637663538626139366165643437333233643830313962663735373063 -37363664363432326637386335386432616135666162363431356561333635616134626136316365 -38323539663966643262656635356334386236393538333034303565393439353931663466373966 -31376466393363396535396263336338306239613961356130613335316132313335636634353461 -35343830633036653337356131386538396139323265653230396664653533636239613239356639 -66316332323538656166393037636563626338386235633230623962373731386230383733366663 -64383064623362626639383030396339623138373935373539643335366531393635316137323266 -61383065353766626463636338383563366532323439616333623730386630656163343536646562 -63343538616139363933316266363531656661633532353765366534383735366664616565623030 -33373561386537613737653764326439366632626537346464363132303133393963313966373163 -32623239326631333338303736613465616435346535336332633163313439656232626239653836 -33343638643562626338353137346365386431663334336231656435373238323535323733623461 -37356533613738663764656466396632366430316433623334666665363463386131303832326564 -62613163366661346435343031373433343236643839396536383966363863393836623265393135 -62303838393539366166383137343737383432333235633230363036656466373837396462393564 -61353535623738616239663535333666323839666133346336646533633735623639376332656464 -35633839633665313033353834383530336164346233643033373336656265363266363534316465 -36346536383535333138313761356561323661643163393831343962633036373265643938336533 -63623135386630356262386563353634623439383631633866366661336336613038623439326264 -39353933643233323863343839656634373732363661373966313839363130646533323737646433 -33626634643964336236646233376235376230616435323930323935373730633663393435646331 -64613839366663663933626537323339613363643839313632313034633135643166323235356632 -66643437353130333831306530653834386137386632626363646563343631333235653065353730 -38323363373665623961306530316562353165656138393134653933303239663636643464306136 -66303234353537633062663735653232366639373030336330336635323065643935323936396333 -64316661323263386664323031333030326534306566333934306361353233373536356435326338 -33303561326233656233366431653237333763326434313162663165633637636262343130636133 -61366635663730653539386366353237633438323966633961303032366136336630623436393035 -62633463313338353432323939343239623630383565636162616139613664613863396165643736 -35376462333938656432373133383838623035653131646366373239306262323539313439343166 -66323830616362363263636663333566386239353765343830343465353265333262313331343434 -30313339303064313263633864623261616130343530336134333661333931663564323461353034 -64663737393430383233633330306565363230383761346362663530646238363631383062646230 -62633432623563353239353737623938663437303061623866363431343436313837343539343436 -30393737346435363563346235633438306339366639613161383763613730353437636537366565 -66396534353230613137326139333435393034346266643364646138666130656532323166623664 -36656631336636393131316361376339383333333837663263343065623937663062346666323464 -31636662386362633337306338633138623434323563643663663835656430333762366631343033 -63343861613365336434653035333132636363656139393065356330376161383039633533393366 -38326630396535383830303437666333623735393131336630663936373465326466616133343735 -34393535326335646265383333646233386434636530613464643266326134396462313762363066 -64303637386261653866623334666532323130393838313034633038356361363239386435656239 -63386163653362386565646561353238653962343033386632336364313630353063356336626338 -63356137616530643434313862643235333063363533666635346661363861383366326533646665 -66316531633163313365653337663633393661313466643331346230613539653663336637353162 -63323730386433626364333632383761323362363664306264336334613766373532386533336662 -65396336313332633539633966313733386335613236666234313664303631326361376232383036 -33663563613864643532363038303832363434386265366637376561373434643239396231643634 -61633935363234383935393333666566383766646461623061613361636531386363363238393465 -66643636633465343033306437626632373734323762666164356231333137376535313831373236 -66373435323339316533383432393766633566306635336166336336643532313430393539336332 -37613666333763396439396466313535383066636237343964306661636337383931636631633463 -64383130613964376462663837383762353634623064313431623266383839333434653537626663 -34363736643937326134616136306236656261313136366635303036646435353863306662353034 -39363266356636623135356232383630373162363330396262323734643564366232333632373739 -36656366633335626131643930366534326164646265643339623539323863393632353838633861 -36363763313039633530623736313239353036666162346432643732343438366265306164383235 -66376536653761383665316665396435646634363436376632646563633163623766366635623731 -66623831393230326566313932653536343838626566623038653534623434313239343763313433 -30376630656638623036336166346133633738303638366238666639653030323832323365353766 -33623261343664323631393166313539623734393438346562613735623636396137333733303235 -34636630643932383536633836343765373234623639643437306532353764386435663665643738 -65313065393534383935303639656534633363303235393761343330386630633734313561376433 -63623464653034336661386435386431306634613239393039386439316538343261306234393534 -30323164363734303037663134663734363636653935336539396330383931353461353966356533 -63316462376262626261623031353063353030363230653966393832626662623963306266393635 -34643565386137663435373639393537626632373535333865366165353839343732353439616265 -37373563326438656134363931623565313565666334323533653264356633353931323661663832 -35396637353163383466653765376132353561343031623039383161623034396134626562666632 -64643737306634613966663362373038396231616632636539663966313364383137326164396533 -38643266316464366161343761633332633036383537653565653139336562666461666465376230 -61656361623264366566656461373863656161393333353933326665333730356333343531653766 -36303964353134396533306630343832303961336239643462353439363962353738663731663033 -38343833323365366662343665633130306537383861613762613763646631626330613937393432 -36393338626661306130353461653565313766623762613564356466393663383339356335306436 -35373636396239653361633764366133363334643232643734383335346635386139383130363938 -35643665316139613639666436633531396537353439633839623063383134376230376235366534 -64393634376633343063646433323439303438303432323335393265366633343332616461386665 -62626132346364373931303936623963663037333665333639333966646431663635303365343630 -36316464346266313166356262346331656432363865643135613964356662366632323736613437 -37336561376334303534653237303964656665643561376237653531653764363463363464313737 -66313335383838333233663162336337373764363131643836363365353039323836346433313139 -33633636306131623335623639666130393535646536346438303435366436623132613562643066 -65623336643463333932383865353232333339343231613964373566653063643662376338316437 -65343466396530386230396236656463396566336230343838366264656161323063336264643364 -31643066366632653962323636346565313833393966343761623066336464333733646238386332 -37393534356333396661353130363462633366343535616338653330393262663066326662366530 -30383536653838333066623063306233613735363665623937373362623337323339373663303933 -64306538636533643337373766346266326238653937613366616161336333636662386238346237 -35623237333131623363376230663132373939366235376631626263393764366661346438383537 -33306233663630323863343839636166303461666135313639356637303435643062303364646132 -64656636396264303962663363376166323834393863393264323464336662393930636533363834 -61363937383766623930623537353037623031363736633332393766323238383464663531653438 -34653933623166326530343632386165666266303438623739356635346163346239643065336537 -33316631613431623030323433306133376237316535306666343433646339313965663565653130 -61303065303530616639383066623534353566366632313465316136613437383765643039313137 -64393333353834353033353532373834343831326535393837323037373136303462373530646430 -62633964626164626139373562616430613662646265396139356561666462376435643031643964 -62316339303561666435663438393838356437646336623664613863303134343261373830303730 -36303732393262363530366563343165386436313833633533623637646464616663353035363235 -31353764353135343033363562313465383563376164323766323735646336353834613264316464 -65376134346437333439376636313531666638313333313963636363353965363239323830663434 -30373234373235326635633063366232333235653935616365656135396465366564313530363161 -33363864303063363363363535323333616661323338323735323135613562366163393366393963 -31363331373661636134333063363731343831333061326431633061663064356466306438646565 -62646431613861393237313138353330323734636666333934666437343665653965316436636464 -64333432313561656339366533363765343365613139626331396636376535326165623735346633 -61666364373465613438643032316362303335333431306166393766663032666665666564313837 -38323232373439376563633939623835656464313361333666623336666461383330663764666166 -34343638646637646338313737336566623937616666626637386639346530616439316635396661 -61656439346231663436386665653031643334393734346163353963633333646239623536333730 -39363731316361633139326637303839633633633738633737326465343664336261646337343831 -33623261363136613935316635663434633962613635616333383238613965646637363735646264 -39626635323532396339653861383137336136346539303931323262663635366432376365623732 -35356639376161323937306136376236643431643863393630633662346531343230643636666633 -33636139323663353835313663656162656264313739663066616663346639326434353762666330 -62303235363964323964663664353265316461373637346230646364393065653763646636303664 -62623735633234303063306531393662326363333234333763613335666233373131303334366665 -30633037323934633232316366663563326562353834643132623031613138323835373933376536 -37323638396561643363663235333632323234303734643637663261393736613738346433333539 -33623865363862653731366466333530343561653661653638353738346438633236343730326239 -35363837393736343632663765626635613865656165396336646538376434653537633037616566 -37393766636661383538386164363630626438616365666532333039663464303835373134333432 -30386439333732666266666331633566326463316438333639336665613933343963363562636335 -31356465303535663233656466616462646161666538346334323335396561656634396631356262 -62323561316261653134363436636333353665613165323935353439666164363163303530353964 -66633536306432323335306530323733623536383366653566653339383161383434613330343934 -63666131333963356336333763386338343437613035313036393766636535386234613665636632 -34383032633931663633386636343233646131393066393766366635343935356538306231336232 -32353836353335363731353234303337343737363064636463613132393563316136626539326264 -30343230343166393466353664613438383462623734363038363962306437663938306235316235 -64373433626366636133373964633131636534333839623033383163613561363430343534313966 -63366561613939623334653434616238616136353665386665346366666236326363376135373363 -36343965343131356662373834366535313963333366633536343930633561626230373736303030 -65373063386232316332343963343631333162376334636665633531313936303232323933313263 -31373639613365396533303431376532643434626538653538356630626532343530663163366232 -39376566353361333639343237653937303161646164613831643235613139373961663566333630 -62663430313832663432306634356464623838303835373962373533313565663735326535326461 -66653939623836666461626631656534616162323433356431303366306635323938373138613031 -66366461353133386436633765386262373736633539643637373237326235633764353132636561 -63343438666361303364333732303738323463353832613438636663363332333132373361633439 -64373334386661626364393664346262653739613134346365636132613538666339393036333163 -64373363643031393062316135336561303665623465336464356165313466633462666331303030 -62626535396634623366313662643066313462636330343237316433646434313737626562346237 -65393161333336386430353938336536336461326132393933343335656131393664356463376631 -38373862363862343839666161663136336534633862303636653931363136303337353664393737 -35316138353361306133636463366463333137616562366135653230343039643365646665636364 -65366564643738356539663331613439656536376263306362366165306230373336386166613865 -33626631623364663062396466626531666238616436326163643533386366633164633262356431 -32313762336561396531386535383434396135633066316334626632633431356363373034663135 -33323166373562326237636265353634623236666364643837353232306263643132343761373230 -63626331393461653662383666646332303632386634313034313664346432656265 +39383462626535653332386339666338643666353535313064613633353230366534396665376235 +3061353662623030623038663063646661616537306531340a346430623235633264346163396661 +61333533646538373838383231353338313262656466613664393132356536396134306265633036 +3861643930326163610a303834613330326536616432646463323731666536383563306162616130 +61613665313665643265393938383664353538653937616230316136643765653632313932643235 +38346639646630396566656165376639383034323439386563376563666662363866303639343230 +37313938613261313962303666666431383864383963333833316531333065323630306165623562 +34303736646663373230303733356536656634656330316362303066303939376230393330376338 +30653666376330303234376132343731376464383061306139643237393463613565613561363335 +63336366613736616336643937326663383161376534626432646437633136646339313566626335 +32303638616466376535373331363564386534396164323538363664623965626635653438613436 +34313466363939343934636635363762626662636462356332646232633539323034646338643139 +34626462303937633032396334643232336363663732636132316136666232623865663434336433 +32363164626635356366323331366634613532666530643233643531613264393639333065396139 +39306132383462633138626231383463623335363130303565336662343933643235303331333335 +30373665303665393532613333383539323332363636366363616236343533373663333864653961 +63613465656465653763313635343263636430653964626662666434666634313664363930656165 +35313265643031653861653237643763663332383266646463633732616666346638333937613339 +61653265663765343764303439663230326234363961303835353535346332666561643266386632 +33633437336639663063373737356536326332336631303834363838346634646535366331616661 +38633235336361633132306330393131643530626661663733366132633939633637656231623438 +36646332363065643930343831316238333566323234643435346530323038626562666165373934 +35376136343566656436343438336363363364303965363533393330393430316566313033313737 +33303536323235346437316636643566306465393031353439666564313165663230643732323563 +38386265633938316462343636636666343638356338393061323134303837393231653435653833 +32616433343465623536653165623435383933656534366664343461363662313736623061616163 +65343664643736316337356134336236663434353635633032626664633333356661646433353566 +39643664366132623363336339313739346634616439396535346533396438646433323538653933 +36373639646566353931643630656432316166306531363562343666353633633063346263626532 +65343437656564633466306166366535623265363566323461626237333862623465653163643463 +38316338353233653131373336306537386638616462366463306464616566623932323633316631 +32386231313430343765306130616361343537393333616466663763633736303131356261626137 +38396634613734356465313865326361343432633066393861383363343262306662316662363330 +61393035666437366635396361313831626138396337616630346339643133366363616264393635 +63336339333862353838383437346165326238313632323862616161303330376464306265306661 +34623530656331623863376661373032333462616462646630303630626631623830343637386133 +61383236636534643932393262316330326161323435316162666566663235336639373431386463 +65363336313439323064333633353236313537613132396632343665616636396632363362366132 +62386538336162333062306136383737323336646163393161316164363561306461373731653334 +64313765313330653762336633396262626361353730353737646534383262663330363335633539 +33646534643231663464376638373934613432623639643061613262383635323133326564346263 +38303863333638353864636436623938326538323765623165633465353433663638363736316432 +66626264646335376432376561336261336633643039663533343562393736303164643833346566 +36656361356130303931666539363231633232393731653262666162396163376230643635636163 +35376163653036383364393638343663383937353239663030396663303431356632346339313931 +34326638333766353763623433393938663732373538323735323166613439663132653961366132 +63333465653035336238633665393134633436386263383862386166633436633834393034623231 +31613362346239313639346262633566373662623261343939313434616336663337306632656434 +37373331333163343035353731363462613436613833386561366430343336386261373835663237 +35643032323065313835353637666663323337626330353138386661383039383162336437343764 +31643037386465623138393236613166366335636633383762623762336535323936363134396436 +66323339383833613363653331643032616438383238643666653938613964393464613434623138 +32343835653332386135653762326238313833653264396266383830633536396131373839343332 +62316132326361646633653732663063366663646366366633353133663963356637663137663263 +62636261336463626439616334633464373566373864393965623166633733643730393664393131 +38636333326465386132636139356339613730366539323762653564616639656630623435656262 +39333331353933396534323562333665373233616135386336353334383436356630373931653237 +66656461353239346435643034643166363765623631373736656233373965373264636566313139 +63613839636565613663623831636465616335633636613437363066353961396631306332636537 +38393230616132316266613331643534363764666637663532353539333661636331623364623433 +31363433326536646439336131633961303737633364623961306466643836653133363235326439 +37616461346433623130616432353035653464366136353062633732313362643033323830626163 +64343866386162313133393637343365336363386563653436336438653836643831333064613461 +32646133636130303166333466363139386330653030326230626136643830666436653036323061 +61306530323666616332373635633132346635376237393237636638346433393131323533643761 +32613230663161643238623865613333323263303831366237386331353935636532343264653134 +61636136636538666461373537376539343136363466343766613961386231313336633438373965 +35666130373163373236396638303738306538653238343838386130663865366137613730616161 +31313330343262393632316635326134646662323535366338656434363162363461646362663761 +63653266653835333762353433633562346164623661653762666135363435376665663437333363 +63646663343030373566383665383864383463366363353630366138366433363362346634383633 +39396235303830626138363539333138363230313463303062383562633234616230333365333831 +66323632656336303231653138636164396532616265346261663839346235643439616466643438 +61653837336532316230313664623233353634643037653138313936653261363537656365323961 +66323563376363396336623164643866373833643138646361383132363537323862336262633766 +35643235613033333530346432376639353266346433386439323830363237666464306535386463 +36303032346234336335613764383165326536643439656563326465366136653964623066343036 +39333439343932346162373030343332373135343064346336353431303330666561623963366465 +38336162303337303235636131656564666536336139393537323831633739643531333838653432 +31633535386230306335616564653832656361643339386638623937326635643465643564623833 +34376633643830316635646131373039653932626231646536386165623435353861353639343734 +65393234646662363435636536346265336337616235376466613865383965316234383434303930 +31363932633765653335393262316363356265643533663762356235386437643238316334356131 +62333836666430616632613962373635633333333135656338323636346432636131613165663830 +34616338386232633066323131663464393963386538393632626265623133363637393537626535 +33303232643465353365626364306663363233333965643464393734626634353065663637363164 +66396261643531386463363931343531663733316336313537316538326436386662616338613765 +36363037323436353035353638613931373364393961336637636438306263376565346234366539 +62353530313335303734643731393636303532343362626361366162663362356463376636646661 +31633161633134626164363430306234366130396235613238646661393034656638313333343766 +39383266623234653338396637336435666134653164383463373734616364323731616462376634 +62643039623931393736383261393131333261623130333363636462343830626435623336666161 +38663936303531353932316430626261316437333536313230353431393262616463653430643136 +32626539316330323664643533653661323038313063623262346264393333343030656365653566 +38303835363837383539333265613865663930633535373230653763323437363535656265363633 +64653230646363373032653032336163353033336539303964663830323935393732393538373835 +34323938616333346139316361376463383531393134316262376332636637353834623937333264 +32633135386232656662383838623238323336633638636631386361306238333035636232376432 +39383634396635343032313164636132363331373830656632626461643464343631363838323233 +61653530616666313635383364633738346134356533336431313935353364376537666635633936 +64303764386538363438393430653236653862653165303631383434613037303639626461623066 +63383966343437646433656531333761393430636236326362656137663861663938323333313238 +34323433653363353565396636623030323633303536303366323030386566313661373063353435 +36346232353838663962353436316133643331383065346536663961373665343338373761653439 +63353463326130356565626164613836393931383236363638643565386230386363363236366536 +35653366393139616362633737616165663438653738636337313861313035353430393962373264 +64303138373339616436623664663638623666336637653662313131646630376134323437616166 +36366330303836396133393764653462316532313434633738316335626430653133633162623734 +31633465623766343566393431306535633033363937376638643865306239316565366162613366 +36323439663836373630643138303235313636663431663738653436646134636230393535623236 +30393036336137346630623035626239313661376539333237313261323438386632303939336566 +31346433623235326630613165323762363835343939623666613638616164626236646433373864 +39623866373238356138336266313135393063373733373332303338343965303538316431663462 +65613766623538633138616632366436316230646162323432313434393938356635663836633036 +62356631626138343436393238306539663532353431366162336130626334633230653638323134 +62313830343730653230346233626633386433313131663233303435373561343036393838633830 +61346135313130323464643531646139393633386461666630613637363833393466353737306530 +33653331346464613431643735396661346431653933346563643531626638653932316238356632 +30343864353431393231386639343237373130376639343236613337643739316132333133373962 +34393962393065363133353338613739613332653365316335306663643331383138323537613866 +35336339613138323031366436616139306339346363663662316265306161343935376539376166 +31316264353636343463636230646431333164626135663833316661363261663332366335326130 +36393438626531663538313931623132303332366538396666303939663061353865633936343162 +61303533383439626636646230623931383636633136336636643436646662396630303637333535 +63623661376130303134633761393835643266326133616236363464326238376466326266633763 +62656637663265363338636137346138616431616139653266386432373461316634326339333932 +31373562366132326635366164353637626334353464323764636463316665373936643030333539 +35646634643339653564303835333163383936386335353139636563363939323632343562633662 +65663033623232393430396139626135326236663463333639353338633736356430323036356537 +35326538616664386233333264313637626436613033643936343063383261653333353339306538 +38313631663263303230363830623235343732626638373535316537343938633632633534386365 +36623336343866383132373162626565313665393964663562653463396130623238643631363838 +38313363666635653134353663653461343135313138636434376362303639356362613639346238 +39343432613164643265333231366139326462313363633234313832386132323239323166613931 +37666135313036313236643261333739306362626236613435343766303162373464306531326333 +66363664306535666235636234346164373064373863646562353331346536613764646633333861 +65323361666430336462656238633735366661666662643632646562366535346432346233663330 +30336261636239613932303032653962376231333536633830646364663135626561336339393364 +30633532616163616363623835306335313163636337353263333836326261643334636666333836 +66653731316639376162386363303766323734653938373866306264376462643038386436323438 +62396136353961613633316661666136323037613863383338613633386536336330333363626237 +66376266623963353439613433633037623135333062636436656663633034343437346262333534 +65333331633037396239653838383462386136623764343436623737643564663261313834333133 +38343934636164323135653435653835333331623864383133313965373065643439623361613631 +62393036356164313665326537626363633839313935663636613434363934616135323036346131 +32363033323862613562353364373035653631366431303865646134316438366266643965623536 +37633835386439343932663861666230653463336332343864363035346135643535343535366135 +61653262363937373036616661333639383362383337316561303163643639306466353334353866 +38653862633732333962616135383234336233383434643264396461646264363134353531356537 +31643030386437303563313033373366366565633435616435643164626334643364613535363232 +64623835393434316131303237393038323930633861396431643833346636353534663362336264 +35326337303036303737336333636432353033366461653435626464646434646461303337643437 +34666635626335333133326663353234393637386330393964633836666364336164633030643534 +33346335643962323866306634383335613463633136636362396464383135666162333866333037 +34383461306332306132303063346634643838363835393564646537663236343432313766336265 +30336537313735386261376365623461303337303430613461373866366665623332343633643439 +62663366323165383463333864653132363632636134333932393333366466356536336666393330 +37346361393831363633623735363032646464303765353935643265373463323366353461316535 +33366265333863613531653663326162383431353932366231396332306234363963653566656130 +32313831653634353865313561363966653633313535363464326664396432623561653738313561 +62626239383562626139623865353039643365613565386337313666386364656537373165663833 +36353264613462393365396562636138663535613433653461303935646438653039653736616564 +66643737643437313366333830303839333965626164646637326331626265323035626464326462 +61343861316532666461336533656430633261393732323232313864623832323662613334386334 +65653837366532613538373138303633346566663330613965633963653864666330346532313038 +37616163626235303364303534383930376262643539633630336162383864386437633162323332 +37393236346334646139373130653438393663393335336331316336386236303535366237616330 +61346663323234626637343532666135633133343234393064323033623832613264623337356139 +39373530366564333437383266333636666363393461303034346566626562393065633732643564 +61393936316330313137386338323461353339323331383533376362323833666635383836333736 +65383738373537633164626237626364623866663031313864633032353137353331393765393339 +62653337623538623664326563316266613263376536653138656632303939626133653964616537 +39613763383938633163323234356139313261613332373861393462323930353066333134306532 +36303061363763316632616134353633323739633665313962623364356562366535326339646635 +62333838373833333133653338366638386566366337303232353537303961373030303464613964 +39626538353033383430333132306637363633353135333232613462656634626438646139653266 +61636365623634396438633731616665623333353735663036303664333461636536313733363232 +61656333613736363039613865383566343336346466366165323737336233303239333163326437 +66373231323333316630653039343731323332656131633131663635636433313732663935303232 +64346664353836383864333163353834313732376638303036323235303034373666653666383264 +35316537646237373535366262656532623331393566316235396631353030663961323238376236 +64303766636134626236646530616331323334613233633932653861333961346138646266363864 +36616563313131633931376230613861623331373839353733353730363234343339303037653663 +63646561346466386131626361326533363739383832383437393033376261323363643966396662 +63323234343931326663636662363837313665383962373965623335346330623735346665306364 +64333239303031666262316239613033303837386164333034616461316237376431643862366231 +63613935653533373530346131346265363565343166656563303164663132346430616234346235 +38343363633063326233306261643466383262346661363933316238303836373831623438653238 +39646438306639383561646666646137356561633131363563663066613263333234636466366636 +63323065346430353834636332383536643061356464333863356666333264356633306131303464 +34343230396339646466663963613333356235316638636463643333346262613637633936623839 +30666430353261363432653162353166326263396333653633663764376535643331336661626666 +64653433666638313838353766376663633564326633376461336234366534656531656638323164 +33636130333032393030633131646532323766383062646431646461353035636338363865366564 +32373664636362613539356665316461326638363631316664626435333238653135376134303035 +32396631666466363338626131393562316638393531633438613133623734323461326232383665 +65356538616538363065326264306236613330303439636232316339363333306238393766363361 +39396634366333636637643136653164623165613733373039366335323061636332363636336330 +61306237646637613961326564626334316334623835616365623132303039373930316137653433 +38306134383738346662 diff --git a/ansible/roles/gatus/README.md b/ansible/roles/gatus/README.md new file mode 100644 index 0000000..129057c --- /dev/null +++ b/ansible/roles/gatus/README.md @@ -0,0 +1,110 @@ +# gatus + +Deploys [Gatus](https://github.com/TwiN/gatus) — health checks, a status page, +and alerting — **as the upstream container image**, on the +`observability` group. + +## Why the container + +Upstream publishes **no binary release assets**. The image is the only artefact +they ship and therefore the only one they test, so it is what `docker run` in +their README gets you, and it is what this role deploys. + +Building from source is entirely possible — their Dockerfile is a bare +`CGO_ENABLED=0 go build`, the Vue dashboard is compiled in via `//go:embed +static` in `web/static.go`, and the sqlite driver is pure-Go +`modernc.org/sqlite` so nothing needs linking. This role did that at first. The +reasons it doesn't now: + +- It produces a binary upstream never ran. +- It means compiling the AWS SDK, gRPC and the Google API libraries on the + smallest box in the estate. On this VPS that load was heavy enough that + unrelated Ansible tasks timed out while it ran. + +The cost of the container is a daemon on the machine whose job is to notice +when everything else breaks. That is a real trade, made deliberately. + +## Pinned by digest, not by tag + +```yaml +gatus_image_digest: "sha256:c5f210d0…" +gatus_image: "ghcr.io/twin/gatus@{{ gatus_image_digest }}" +``` + +A tag is mutable — `v5.36.0` can be repushed — so pinning the tag alone is a +weaker promise than it looks. The digest is a content address: if it resolves, +it is byte-for-byte the image reviewed here, and `docker compose pull` either +fetches exactly that or fails. `gatus_version` is kept alongside it purely so a +human can read which release it is; **the two must be updated together.** + +## What `FROM scratch` means for running it + +The image has no `/etc/passwd`, so there is no user to drop to by name and the +default is root. The compose file runs it by numeric id (`10001:10001`) and the +data directory on the host is owned to match. Everything else is locked down to +approximate what the systemd unit used to do natively: + +| systemd | compose | +|---|---| +| `ProtectSystem=strict` | `read_only: true` | +| `NoNewPrivileges=true` | `security_opt: [no-new-privileges:true]` | +| `CapabilityBoundingSet=` | `cap_drop: [ALL]` | +| `AmbientCapabilities=CAP_NET_RAW` | `cap_add: [NET_RAW]` (for `icmp://`) | + +## Configuration is a directory, not a file + +`GATUS_CONFIG_PATH` points at `/opt/gatus/config`, and Gatus merges every `*.yaml` +underneath it — maps deep-merge, lists append. This role owns exactly one file: + +``` +/opt/gatus/config/00-base.yaml web, storage, ui, alerting, security (this role) +/opt/gatus/config/endpoints/*.yaml one file per service (gatus_endpoint) +``` + +**A primitive defined in two files is ambiguous and upstream refuses it.** So +anything that is not a list belongs in `00-base.yaml` and nowhere else. Endpoints +are lists, so each service's file appends cleanly — the same shape as +`caddy_site`, where each service contributes its own vhost. + +## Pull and push + +Gatus polls. For anything with a reachable HTTP or TCP surface that is the +better check, because it tests the path a user actually takes. The monitoring +host joins the headscale mesh via `infra/920`, so internal boxes are reachable +by MagicDNS name and can be polled directly rather than having to report in. + +For state with no pollable surface — ZFS pool health, UPS mains status, disk +usage, backup freshness — Gatus has **external endpoints**, a push API: + +``` +POST /api/v1/endpoints/{group}_{name}/external?success=true&error=&duration= +Authorization: Bearer +``` + +with `heartbeat.interval` to alert when nothing reports in. That is the same +shape as the generic `healthcheck_push_url` already wired into every service +role, so those scripts need a URL, a POST, and an auth header — not a rewrite. + +## Variables + +See `defaults/main.yml`. The ones that matter: + +| Variable | Default | Note | +|---|---|---| +| `gatus_version` / `gatus_image_digest` | `v5.36.0` / `sha256:c5f210d0…` | must move together | +| `gatus_bind_address` | `127.0.0.1` | **never bind publicly** — the push API shares this listener | +| `gatus_storage_type` | `sqlite` | `memory` loses all history on restart | +| `gatus_alerting` | `{}` | pass-through; any provider Gatus supports | +| `gatus_allow_icmp` | `true` | adds back `NET_RAW` for `icmp://` checks | + +`gatus_alerting` empty is valid and is the current state: every condition is +still evaluated and recorded, there is just nowhere to shout yet. + +## Verifying + +```bash +docker ps --filter name=gatus +docker logs gatus --tail 50 +curl -s localhost:8080/health +ls /opt/gatus/config/endpoints/ +``` diff --git a/ansible/roles/gatus/defaults/main.yml b/ansible/roles/gatus/defaults/main.yml new file mode 100644 index 0000000..95c41ae --- /dev/null +++ b/ansible/roles/gatus/defaults/main.yml @@ -0,0 +1,83 @@ +--- +# Gatus, deployed the way upstream distributes it: the container image. +# +# Upstream publishes NO binary release assets - the image is the only artefact +# they ship, and therefore the only artefact they test. Building from source is +# possible (`go build` alone is enough; the Vue dashboard is compiled in via +# `//go:embed static`, and CGO_ENABLED=0 works because the sqlite driver is +# pure-Go modernc.org/sqlite) but it produces a binary upstream never ran, and +# it means a full compile of the AWS SDK and gRPC on the smallest box in the +# estate. + +# ── Image ──────────────────────────────────────────────────────────────────── +gatus_version: "v5.36.0" +# Pinned by DIGEST, not by tag. A tag is mutable - `v5.36.0` can be repushed - +# so pinning the tag alone is a weaker promise than it looks. The digest is the +# content address: if it resolves, it is byte-for-byte the image reviewed here. +# Both must be updated together; the tag is kept only so humans can read it. +gatus_image_digest: "sha256:c5f210d095fa78e6efaa20ffeb14803f2ba4f10615e16a6d12087697149617f0" +gatus_image: "ghcr.io/twin/gatus@{{ gatus_image_digest }}" + +# ── Paths (host side) ──────────────────────────────────────────────────────── +gatus_dir: /opt/gatus +gatus_config_dir: "{{ gatus_dir }}/config" +gatus_data_dir: "{{ gatus_dir }}/data" + +# Gatus merges every *.yaml under GATUS_CONFIG_PATH and its subdirectories: +# maps deep-merge, lists append. That is why this role ships a config DIRECTORY +# rather than one file - each service contributes its own endpoint file, the +# same way each service contributes a vhost through `caddy_site`. +# +# Primitives must be defined exactly once across all files or the merge is +# ambiguous, so everything that is not a list lives in the base file and +# nowhere else. +gatus_base_config_file: "00-base.yaml" +gatus_endpoints_dir: "{{ gatus_config_dir }}/endpoints" + +# ── Identity ───────────────────────────────────────────────────────────────── +# The image is FROM scratch, so it has no /etc/passwd and no user to drop to by +# name. Run it by numeric uid/gid instead, and own the data volume to match. +gatus_uid: 10001 +gatus_gid: 10001 + +# ── Web ────────────────────────────────────────────────────────────────────── +gatus_port: 8080 +# HOST-side address the container's port is published on. Gatus itself always +# binds 0.0.0.0 inside the container - see the note in config.yaml.j2. Never +# publish this on 0.0.0.0: the external-endpoint push API shares the dashboard's +# listener, and Caddy is what should be in front of both. +gatus_bind_address: "127.0.0.1" +gatus_ui_title: "Status" +gatus_ui_header: "Status" + +# ── Storage ────────────────────────────────────────────────────────────────── +# sqlite, not memory: history has to survive a restart, or the dashboard lies +# about uptime after every deploy. Path is INSIDE the container. +gatus_storage_type: sqlite +gatus_storage_path: "/data/gatus.db" +gatus_storage_caching: true + +# ── Alerting ───────────────────────────────────────────────────────────────── +# Pass-through: rendered verbatim under `alerting:`, so any provider Gatus +# supports works without touching this role. Empty means "check and record, +# alert nowhere" - valid, and the default until a provider is chosen. +gatus_alerting: {} +gatus_default_alerts: [] + +# ── Security ───────────────────────────────────────────────────────────────── +# gatus_basic_auth: {username: admin, password-bcrypt-base64: "..."} +gatus_basic_auth: {} + +gatus_maintenance: {} +gatus_log_level: INFO + +# ── Self-check ─────────────────────────────────────────────────────────────── +# Gatus panics on a config with no endpoints, so the role always ships one. +# Turning this off is only safe once another file in endpoints/ provides one. +gatus_self_check: true + +# ── ICMP ───────────────────────────────────────────────────────────────────── +# Gatus supports icmp:// endpoints. Raw ICMP needs CAP_NET_RAW, which the +# container would get free only if it ran as root; it does not. Set false if +# you never use icmp:// checks and want the capability dropped entirely. +gatus_allow_icmp: true diff --git a/ansible/roles/gatus/handlers/main.yml b/ansible/roles/gatus/handlers/main.yml new file mode 100644 index 0000000..25ad865 --- /dev/null +++ b/ansible/roles/gatus/handlers/main.yml @@ -0,0 +1,5 @@ +--- +- name: Restart gatus + ansible.builtin.command: + cmd: docker compose up -d --force-recreate + chdir: "{{ gatus_dir }}" diff --git a/ansible/roles/gatus/tasks/configure.yml b/ansible/roles/gatus/tasks/configure.yml new file mode 100644 index 0000000..94f71f0 --- /dev/null +++ b/ansible/roles/gatus/tasks/configure.yml @@ -0,0 +1,54 @@ +--- +- name: Assert Docker is available + ansible.builtin.command: docker --version + register: gatus_docker_check + changed_when: false + failed_when: gatus_docker_check.rc != 0 + +- name: Create the gatus directories + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + owner: "{{ item.owner }}" + group: "{{ item.group }}" + mode: "{{ item.mode }}" + loop: + - {path: "{{ gatus_dir }}", owner: root, group: root, mode: "0755"} + # Config is root-owned and world-unreadable: it holds external-endpoint + # tokens. The container mounts it read-only and reads it as gatus_uid, so + # that id needs group access - hence the group ownership below. + - {path: "{{ gatus_config_dir }}", owner: root, group: "{{ gatus_gid }}", mode: "0750"} + - {path: "{{ gatus_endpoints_dir }}", owner: root, group: "{{ gatus_gid }}", mode: "0750"} + # Data is the one path the container writes to, so it must be owned by the + # numeric id the container runs as. `read_only: true` makes everything else + # in the container immutable. + - {path: "{{ gatus_data_dir }}", owner: "{{ gatus_uid }}", group: "{{ gatus_gid }}", mode: "0750"} + +- name: Write the base gatus configuration + ansible.builtin.template: + src: config.yaml.j2 + dest: "{{ gatus_config_dir }}/{{ gatus_base_config_file }}" + owner: root + group: "{{ gatus_gid }}" + mode: "0640" + notify: Restart gatus + +# Without this the container crash-loops on an empty config. See the template. +- name: Write the gatus self-check endpoint + ansible.builtin.template: + src: endpoint-self.yaml.j2 + dest: "{{ gatus_endpoints_dir }}/00-self.yaml" + owner: root + group: "{{ gatus_gid }}" + mode: "0640" + when: gatus_self_check | bool + notify: Restart gatus + +- name: Write the docker compose file + ansible.builtin.template: + src: docker-compose.yml.j2 + dest: "{{ gatus_dir }}/docker-compose.yml" + owner: root + group: root + mode: "0644" + notify: Restart gatus diff --git a/ansible/roles/gatus/tasks/main.yml b/ansible/roles/gatus/tasks/main.yml new file mode 100644 index 0000000..e1b67bf --- /dev/null +++ b/ansible/roles/gatus/tasks/main.yml @@ -0,0 +1,5 @@ +--- +# import, never include: dynamic includes are opaque to --list-tasks, which is +# the primary verification tool in this repo. +- ansible.builtin.import_tasks: configure.yml +- ansible.builtin.import_tasks: service.yml diff --git a/ansible/roles/gatus/tasks/service.yml b/ansible/roles/gatus/tasks/service.yml new file mode 100644 index 0000000..bfcca3b --- /dev/null +++ b/ansible/roles/gatus/tasks/service.yml @@ -0,0 +1,35 @@ +--- +# Pulling by digest means this either fetches exactly the reviewed image or +# fails. There is no "latest wins" path. +- name: Pull the pinned gatus image + ansible.builtin.command: + cmd: docker compose pull + chdir: "{{ gatus_dir }}" + register: gatus_pull + changed_when: "'Downloaded newer image' in gatus_pull.stderr or 'Pull complete' in gatus_pull.stderr" + +- name: Start gatus + ansible.builtin.command: + cmd: docker compose up -d --remove-orphans + chdir: "{{ gatus_dir }}" + register: gatus_up + changed_when: "'Started' in gatus_up.stderr or 'Created' in gatus_up.stderr or 'Recreated' in gatus_up.stderr" + +- name: Flush handlers so a config change is live before it is verified + ansible.builtin.meta: flush_handlers + +- name: Wait for gatus to answer + ansible.builtin.uri: + url: "http://{{ gatus_bind_address }}:{{ gatus_port }}/health" + status_code: [200, 404] + register: gatus_health + until: gatus_health.status in [200, 404] + retries: 12 + delay: 5 + +- name: Assert gatus is running + ansible.builtin.assert: + that: + - gatus_health.status in [200, 404] + fail_msg: "gatus did not come up on {{ gatus_bind_address }}:{{ gatus_port }} - check `docker logs gatus`" + success_msg: "gatus is answering on {{ gatus_bind_address }}:{{ gatus_port }}" diff --git a/ansible/roles/gatus/templates/config.yaml.j2 b/ansible/roles/gatus/templates/config.yaml.j2 new file mode 100644 index 0000000..67d6244 --- /dev/null +++ b/ansible/roles/gatus/templates/config.yaml.j2 @@ -0,0 +1,56 @@ +# {{ gatus_base_config_file }} — managed by Ansible (roles/gatus) +# +# BASE CONFIGURATION ONLY. +# +# Gatus merges every *.yaml under GATUS_CONFIG_PATH: maps are deep-merged and +# lists are appended, but a primitive defined in two files is ambiguous and +# upstream refuses it. So `web`, `storage`, `ui`, `alerting` and `security` are +# set here and MUST NOT appear in any other file in this directory. +# +# Endpoints are lists, so they append cleanly. Each service drops its own file +# into endpoints/ via the gatus_endpoint role - the same shape as caddy_site. + +web: + # 0.0.0.0 is the CONTAINER's interface, not the host's. This must not be + # 127.0.0.1: that is the container's own loopback, which docker-proxy cannot + # reach, and gatus comes up healthy while the published port refuses every + # connection. + # + # The isolation comes from the port mapping in docker-compose.yml, which + # publishes to {{ gatus_bind_address }} on the host. Caddy fronts that, and it + # matters because the external-endpoint push API shares this listener with the + # dashboard. + address: 0.0.0.0 + port: {{ gatus_port }} + +storage: + type: {{ gatus_storage_type }} +{% if gatus_storage_type != 'memory' %} + path: {{ gatus_storage_path }} +{% endif %} + caching: {{ gatus_storage_caching | bool | lower }} + +ui: + title: {{ gatus_ui_title }} + header: {{ gatus_ui_header }} +{% if gatus_alerting %} + +alerting: +{{ gatus_alerting | to_nice_yaml(indent=2) | indent(2, true) }} +{% else %} + +# No alerting provider is configured yet. Gatus still evaluates every condition +# and records every result; it simply has nowhere to shout. Setting +# `gatus_alerting` is the single change needed to wire one up. +{% endif %} +{% if gatus_basic_auth %} + +security: + basic: +{{ gatus_basic_auth | to_nice_yaml(indent=2) | indent(4, true) }} +{% endif %} +{% if gatus_maintenance %} + +maintenance: +{{ gatus_maintenance | to_nice_yaml(indent=2) | indent(2, true) }} +{% endif %} diff --git a/ansible/roles/gatus/templates/docker-compose.yml.j2 b/ansible/roles/gatus/templates/docker-compose.yml.j2 new file mode 100644 index 0000000..67d7573 --- /dev/null +++ b/ansible/roles/gatus/templates/docker-compose.yml.j2 @@ -0,0 +1,44 @@ +# Managed by Ansible (roles/gatus) +services: + gatus: + image: {{ gatus_image }} + container_name: gatus + restart: unless-stopped + + # The image is FROM scratch: no /etc/passwd, so there is no user to drop to + # by name and the default is root. Run it by numeric id instead. + user: "{{ gatus_uid }}:{{ gatus_gid }}" + + ports: + # Loopback on purpose. Caddy fronts this, and the external-endpoint push + # API is served from the same listener as the dashboard - publishing + # 0.0.0.0 would put both straight on the public internet. + - "{{ gatus_bind_address }}:{{ gatus_port }}:{{ gatus_port }}" + + environment: + GATUS_CONFIG_PATH: /config + GATUS_LOG_LEVEL: "{{ gatus_log_level }}" + + volumes: + - {{ gatus_config_dir }}:/config:ro + - {{ gatus_data_dir }}:/data + + # Hardening. The systemd unit this replaced got most of it from + # ProtectSystem/NoNewPrivileges/etc; these are the container equivalents. + read_only: true + security_opt: + - no-new-privileges:true + cap_drop: + - ALL +{% if gatus_allow_icmp %} + cap_add: + # icmp:// endpoints need raw sockets. Dropped above with ALL, added back + # explicitly so the grant is visible rather than inherited from root. + - NET_RAW +{% endif %} + + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" diff --git a/ansible/roles/gatus/templates/endpoint-self.yaml.j2 b/ansible/roles/gatus/templates/endpoint-self.yaml.j2 new file mode 100644 index 0000000..197ae56 --- /dev/null +++ b/ansible/roles/gatus/templates/endpoint-self.yaml.j2 @@ -0,0 +1,25 @@ +# 00-self.yaml — managed by Ansible (roles/gatus) +# +# Gatus refuses to start with no endpoints at all: +# panic: error parsing config: configuration should contain at least one +# endpoint or suite +# +# So the role ships one. Checking its own listener is not as circular as it +# looks: it proves the config directory parsed, the container is serving, and +# the storage backend accepted a write. If this row is missing from the +# dashboard, the dashboard is not telling you the truth about anything else. +# +# It is also what keeps `gatus` deployable before any service has contributed +# an endpoint file of its own. + +endpoints: + - name: gatus + group: infrastructure + url: "http://localhost:{{ gatus_port }}/health" + interval: 60s + conditions: + - "[STATUS] == 200" +{% if gatus_default_alerts %} + alerts: +{{ gatus_default_alerts | to_nice_yaml(indent=2) | indent(6, true) }} +{% endif %} diff --git a/ansible/services/gatus/deploy_gatus_playbook.yml b/ansible/services/gatus/deploy_gatus_playbook.yml new file mode 100644 index 0000000..704ada9 --- /dev/null +++ b/ansible/services/gatus/deploy_gatus_playbook.yml @@ -0,0 +1,83 @@ +--- +# Gatus: health checks, status page and alerting for the whole estate. +# +# Built from source and run under systemd - upstream publishes no binaries, and +# their Dockerfile shows the runtime needs nothing but the static binary and a +# CA bundle. See roles/gatus/README.md. +- name: Deploy Gatus on the observability host + hosts: observability + become: yes + roles: + - gatus + +# The dashboard is bound to loopback; Caddy publishes it. +# +# Auth is done HERE, at the edge, and not with Gatus's own `security.basic`. +# Gatus's security middleware protects exactly four routes (api/api.go): +# +# /api/v1/endpoints/statuses +# /api/v1/endpoints/:key/statuses +# /api/v1/suites/statuses +# /api/v1/suites/:key/statuses +# +# Everything else is registered on the UNPROTECTED router, including +# /api/v1/config, every badge, and - the part that matters - +# /api/v1/endpoints/:key/uptimes/:duration and .../response-times/:duration/history, +# which return real per-endpoint data to anyone who can guess a key. Keys are +# just "_". So Gatus's own auth makes the dashboard render empty +# while leaving the data readable, which is worse than it looks. +# +# The one route that must NOT sit behind basic auth is the external-endpoint +# push API. It authenticates with `Authorization: Bearer `, and basic +# auth wants `Authorization: Basic <...>` - same header, two schemes, and the +# push clients lose. It is not actually unauthenticated: the handler 401s on a +# missing prefix, an empty token, or a token that does not match that endpoint's +# own. Upstream's comment on the route says exactly that. +- name: Publish the Gatus status page through Caddy + hosts: observability + become: yes + tasks: + - name: Require the dashboard credentials to be set + ansible.builtin.assert: + that: + - gatus_dashboard_username is defined + - gatus_dashboard_username | length > 0 + - gatus_dashboard_password_hash is defined + - gatus_dashboard_password_hash.startswith('$2') + fail_msg: >- + gatus_dashboard_username and gatus_dashboard_password_hash must be in + the vault. Generate the hash on the observability host, which runs + Caddy natively, so the bcrypt cost and format match what verifies it: + caddy hash-password --plaintext 'your-password' + then: ansible-vault edit group_vars/all/vault.yml + + - name: Configure the Caddy vhost for Gatus + ansible.builtin.include_role: + name: caddy_site + vars: + caddy_site_name: gatus + caddy_site_domain: "{{ subdomains.gatus }}.{{ root_domain }}" + # caddy_site_body rather than caddy_site_upstream + caddy_site_basic_auth, + # because that pair applies auth to the whole site with no way to carve + # out the push path. `handle` blocks are mutually exclusive and first + # match wins, so the push API gets a route of its own. + caddy_site_body: | + @push { + path /api/v1/endpoints/*/external + method POST + } + + # Push API: Bearer-authenticated by Gatus itself. No basic auth here, + # or the Authorization header collides. + handle @push { + reverse_proxy 127.0.0.1:{{ gatus_port | default(8080) }} + } + + # Everything else: the dashboard, the config endpoint, the badges and + # the uptime/response-time history. + handle { + basic_auth { + {{ gatus_dashboard_username }} {{ gatus_dashboard_password_hash }} + } + reverse_proxy 127.0.0.1:{{ gatus_port | default(8080) }} + }