From bf3d21fef773b8b99698328122bbfdef7c4bd4a6 Mon Sep 17 00:00:00 2001 From: counterweight Date: Mon, 14 Sep 2026 10:30:28 +0200 Subject: [PATCH] uptime kuma: remove every live reference, repoint the probes to Gatus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing in the repo pushes to, authenticates against, or is gated by Uptime Kuma any more. ── The sixth instance of the banner bug ──────────────────────────────────── memos had `Restart memos` guarded by `uptime_kuma_enabled`, because the deprecation banner was placed immediately above it and swept it in. It is a HANDLER, so every memos config change since 2026-09-11 applied to disk and silently never restarted the service. Ungated. That is the same failure found in forgejo-runner's self-assert, phoenixd's timer enable, mempool's three timer enables, fulcrum's restart handler and bitcoind's restart handler. Every guard was read and asked "monitoring or deployment?" before being deleted, which is the only reason this was caught. ── What was removed ──────────────────────────────────────────────────────── 30 uptime_kuma_enabled guards across 7 unconverted service playbooks, and the 29 Kuma monitor-creation tasks they gated (embedded Python that drove the Kuma API, temp credential files, cleanup) 7 dead uptime_kuma_api_url definitions 7 stale DEPRECATED banners uptime_kuma_enabled and subdomains.uptime_kuma from group_vars/all healthcheck_push_urls from the vault - 30 push tokens services/ntfy/setup_ntfy_uptime_kuma_notification.yml -> archive/ The explanatory comments in the six converted roles are KEPT on purpose. They record why a handler is ungated, and deleting the explanation invites someone to helpfully re-add the guard. ── The probes moved rather than died ─────────────────────────────────────── Eight per-service health checks were still pushing to Kuma. They are not superseded by infra/401: that answers "is the unit running", these answer "does the service actually respond" - an RPC call to bitcoind, a TCP connect to Fulcrum's Electrum port, an HTTP fetch from Mempool's backend. A process can be perfectly `active` and useless. So they were repointed, not deleted. Gatus external endpoints take a POST with a bearer token and success=true|false where Kuma took a GET with ?status=up, so report() now maps up/down to true/false internally and no call site changed. Registered by infra/403 as the `probe` group, one token per host. Two bugs fixed while in there: * forgejo-runner's check only ever reported SUCCESS - it exited before pushing when the runner was down, so a failure was invisible until the heartbeat window expired. Reporting the failure is the entire point of a check. * All six healthcheck .service units were mode 0644 and now carry a bearer token. They are 0600. Verified: 91 endpoints, 91 UP, 0 DOWN. Every probe triggered by hand and confirmed arriving. Zero Kuma URLs left in the vault, zero live references in any playbook or role. Still standing, deliberately: the Kuma container on watchtower, its Caddy vhost, and the uptime.contrapeso.xyz DNS record. Turning the service off is a separate decision from removing the code that talked to it. Co-Authored-By: Claude Opus 5 (1M context) --- ansible/group_vars/all/main.yml | 5 - ansible/group_vars/all/vault.yml | 409 +++++++----------- .../infra/403_service_probe_registration.yml | 47 ++ ansible/roles/bitcoin_knots/defaults/main.yml | 3 + .../roles/bitcoin_knots/tasks/healthcheck.yml | 2 +- .../templates/healthcheck.service.j2 | 1 + .../bitcoin_knots/templates/healthcheck.sh.j2 | 11 +- ansible/roles/datum_gateway/defaults/main.yml | 3 + .../roles/datum_gateway/tasks/healthcheck.yml | 2 +- .../templates/healthcheck.service.j2 | 1 + .../datum_gateway/templates/healthcheck.sh.j2 | 11 +- .../roles/forgejo_runner/defaults/main.yml | 3 + .../forgejo_runner/tasks/healthcheck.yml | 2 +- .../templates/healthcheck.service.j2 | 2 + .../templates/healthcheck.sh.j2 | 38 +- ansible/roles/fulcrum/defaults/main.yml | 3 + ansible/roles/fulcrum/tasks/healthcheck.yml | 2 +- .../fulcrum/templates/healthcheck.service.j2 | 1 + .../roles/fulcrum/templates/healthcheck.sh.j2 | 11 +- ansible/roles/mempool/defaults/main.yml | 2 + ansible/roles/mempool/tasks/healthcheck.yml | 2 +- .../templates/healthcheck-backend.sh.j2 | 11 +- .../templates/healthcheck-frontend.sh.j2 | 11 +- .../templates/healthcheck-mariadb.sh.j2 | 11 +- .../mempool/templates/healthcheck.service.j2 | 1 + ansible/roles/phoenixd/defaults/main.yml | 3 + ansible/roles/phoenixd/tasks/healthcheck.yml | 2 +- .../phoenixd/templates/healthcheck.service.j2 | 1 + .../phoenixd/templates/healthcheck.sh.j2 | 11 +- .../deploy_bitcoin_knots_playbook.yml | 3 +- .../deploy_datum_gateway_playbook.yml | 3 +- .../deploy_forgejo_runner_playbook.yml | 3 +- .../forgejo/deploy_forgejo_playbook.yml | 121 ------ .../fulcrum/deploy_fulcrum_playbook.yml | 3 +- .../headscale/deploy_headscale_playbook.yml | 120 ----- .../lnbits/deploy_lnbits_playbook.yml | 121 ------ .../services/memos/deploy_memos_playbook.yml | 134 ------ .../mempool/deploy_mempool_playbook.yml | 9 +- .../deploy_ntfy_emergency_app_playbook.yml | 125 ------ .../deploy_personal_blog_playbook.yml | 121 ------ .../phoenixd/deploy_phoenixd_playbook.yml | 3 +- .../deploy_vaultwarden_playbook.yml | 126 ------ ansible/site.yml | 4 + .../setup_ntfy_uptime_kuma_notification.yml | 0 44 files changed, 337 insertions(+), 1171 deletions(-) create mode 100644 ansible/infra/403_service_probe_registration.yml rename {ansible/services/ntfy => archive/uptime_kuma}/setup_ntfy_uptime_kuma_notification.yml (100%) diff --git a/ansible/group_vars/all/main.yml b/ansible/group_vars/all/main.yml index 0ff8ef8..316b178 100644 --- a/ansible/group_vars/all/main.yml +++ b/ansible/group_vars/all/main.yml @@ -6,7 +6,6 @@ root_domain: contrapeso.xyz # Uptime Kuma was decommissioned on 2026-09-11. The monitoring blocks in the # playbooks are kept deliberately — the check logic is meant to be rewired to # whatever replaces it. This flag keeps them inert until then. See archive/uptime_kuma/. -uptime_kuma_enabled: false # age recipient for all backup artefacts age_backup_recipient: "age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza" @@ -28,10 +27,6 @@ subdomains: # Monitoring gatus: status ntfy: ntfy - # Uptime Kuma IS still running and this subdomain DOES resolve - # (164.92.239.72, HTTP 302). Only the Ansible code and the vault credentials - # were retired. A comment here previously claimed the opposite. - uptime_kuma: uptime # VPN infrastructure (spacey) headscale: headscale diff --git a/ansible/group_vars/all/vault.yml b/ansible/group_vars/all/vault.yml index d68ae8b..eafc633 100644 --- a/ansible/group_vars/all/vault.yml +++ b/ansible/group_vars/all/vault.yml @@ -1,254 +1,157 @@ $ANSIBLE_VAULT;1.1;AES256 -63323431376238353966386463626539656230326233323861656165386335383832316631353236 -3731396264313366313166653861313736666435346537630a303363346131366433313264626135 -30633836613636393833333239666364623962383763343434353463343739633033383433306664 -6339616335366562310a393463663239333530633034373462313537376266393937373033346233 -62333331613063356535353134613538636663383166353731616633336366653864656333356330 -37323139616436353065363866633139323764336432656663326236636466356432333132656530 -39396566656161373365653738316162653134393234613130356561663464623564316161383566 -32356161646631623134313733343030353064666635346134653361366635316362316564323562 -38623236383137383665623934336631636537343361656539393538346665613234333638346466 -61653434653430636534636137613236326330356234616137303130666363323861623230626662 -37363839333334373561323863353563303861636137613338303736343064663232663064303466 -62396466363966303339643938653930313163323561633835396562636363633633646163363461 -61636331316161386630393766326431316239326563633033376538323330303863646162303332 -62306439666232626533306238343338343938626461363939326363333137306430643033373363 -65393164626166366663656637373330633939326361653261336339393135363934376164303537 -33326631396161393936656563336636643132666232343035633465383632613661633135343165 -33656162656334353636303130613231393835626166633666316561343165333138376439356539 -35356461353832643166613437343633346563393636323631353034366233323566613039343030 -65356438656664366434343638643963396563663434623961663432653334646639653435343262 -31366465366337633638393061643139643138316536396333653035613132623230646561373465 -65366332386565623161303563386237666538633433386438623535386564633937386434343264 -39653335343235386439383964356664313339356531323732363362643566363964653634393039 -37643338316264633733383531633934373132643034653433316438303962356139376364306466 -37613232636365623732313233303766373566623162303965353163373131363763346135313230 -39316466613136323039373765613336333835323465323737323535393736366433343664616539 -32303839376538393238613763366433346433643436613662383062306366643561626164363430 -38353738653734393237663765613733633230343965363732643336333337646438623562303561 -34373031623466343539663738613561346665316163623631643236633236633765616361623764 -35343465346435656533336464393633616663343239343162343664363763373736656431663433 -62306565643862613065386265613036326336353130633530373166343966343033346665346235 -37333761613437393236393935646239373930613239323639616564383336376664386532366236 -31313862363232656537613733666239373433343835633964333164633335656436373766353838 -36303234353333333233313531396464336262663864633638306236343633336432333737376539 -36623439336133356362626632663966336162646636393932353537356638336337346663303230 -66623964656133333534313231656337336463346363363635396135393062643736336133393134 -66376233376161373561386531633639376137643732613535333066646666616133623963323137 -63356565383430656535633639643363653232323737363566663832633931616230343132663563 -37633435626266393937373831366632643535623634386131343065353036653163326464666364 -39613732653739643436303665363132336235333339653630353335646432306432643235376139 -33643935366365383232646234313436383534353130633039376562363939363033643830303936 -64376131346631306263363137316435343661393562386638363636336261623831616232313934 -30373133633034626430313936373537323366626562323239623538306130623135333562333333 -33303761623232666165323438643364313530316563396466653331636531366166633964353033 -61633861363833613264353036383766343233646439393764636664353938656335623330323062 -64306565626630326632383963353266383464393530623639333836663739306132346336373936 -36636532623032666433303339643062383663646536636663383662646237323336386366386138 -31643930356330343138323132613238333837356137643061346364653134346165383661366663 -31643630353631656131633430323838383233323936623661613466663761313264383338373636 -63323062646334666665633662366364383434393561313863653862633533356166396336303133 -61383637326535656539656538353238326337353138616533313534643131346163356533396331 -30303031656133623735366163323664626362356365663730306438653132396461393539386138 -65343865633765306337663830343931376265666362653361616662666665386138313462633432 -32623436643239363437373161353162626663623332373163656133613465333139356564383130 -36653433653535326433313337636635336132663636383764653237356432313362393535306630 -32663738663835336131353737303262313966663264303264313864353663323733653334313263 -32333866613335666436303236393536633835653837363133666437303736303630633939616232 -65316337646665343062353864623839613263356335353938386136666166363565303062633331 -39633761316630656662636136393834626564616139396336663363373931666132383062626536 -61376462333131633634383130393765353037653536653837373032663636623031316530613961 -63633861613366373235653735616333303262333765636531353734633664383766623261636238 -32623132336137666366643035306562643833333537633666343637366230623366666230366638 -37333238336435636235333063636538666635666438383239336164613536666262343132646634 -38633132343966663564326138346561313731623966356435653937383431396261313138656261 -65326265323333366538656335306431396666643663313561663964626436356464356336323661 -66663763396166313831623534353131633566346634613738316634653935373038303730336662 -34656634343834623930623833386632306136323738646535396539643463393861343832333065 -35333464316163626130366233663439633435333462353534616530316464666537613436386637 -37363939333834626638396337636234343561313565613636643463393132626466333632636463 -31326261656537616130613634323736313132633361653162666631303965613036653434653236 -33313037313034356139353261666138386636663637353831666330616563323536303834623863 -30346539376536396337313561396561363237613865623533633063613936316162353138306164 -64646634396236646535633135613036376538343964363663616234666532386165643938396436 -36333832313136363965383334323430313630663336343562626365633461366133653931376139 -38613364383236393634386436643733333433336563306138356337636337623239646164306539 -61336433383339353861313131343166353265393132356631366230343438646239323730613865 -35333438326236353264386436373338303532646336333161636232393235653830303237323137 -36323464303634663839326230613539376238323931663137616363663434373662636535373266 -38363261653937633363316665613130386135626135333662356261323462303939383962363462 -37663333386336626261663464393561666135633537616365376665393664346466633932656635 -65343762656162613831643265633562373865616662313631313034363964626538383633656363 -61363639343663643638353935643635643663636139313963303739343830613336343661663733 -32323463626264663639396130343930383162373133386232613635633934393334626631303561 -64346636636330623538343534663037363635323138343566303664613337346638353637306336 -36386432326661316633303764323761303363646337303539633831666665373833666663333766 -62363461616462383663316434373561653063323062356131643930633838336364643065613838 -66386565373635396138623832646137636236646335386463326265326565353566346136306564 -36323666333331396636353762363866616239343362313431313765373334386637633366623936 -31333432663331623232373330306430626264333761373362303365656262303164656439623637 -36663065376137336435646232323262646439623266333534623766353035376663353535333137 -38373335313833633239313963613439316433643832653938386434323666333263373437663337 -32613661666664366636616562396232366237333237316430346565653066623561636263343265 -34313137643933366239353836383265633030373636393232663534393036343130633438643331 -36643236636232323532363036646563646436613630323638356534373831373737386139393664 -62616133323861333265626665616331333665356666643734666537356565393334373036663330 -66346361363361303538663936616664663864316530303338316634616636353235353161366135 -33363630626566336363336630656331343437393666353262396137386163633134343238623361 -61306163333831626164326461616630636266633666383765356539656536363133393537333263 -66646635343437643762373432333164626334333730373762336565636533333132363965636334 -64316232623437353263656131633335396166616133653234623432386234666531356431373333 -63316462643464626362646463323935303765643061343535393065333961303931663564353735 -33336565383332303637383538316237343430626432393533376166323565303263643435313136 -30626165623765346430373165323030386635376338666235306534643730656564653531633765 -36663864396638666165303837646236386434326166663365356164353036646464363132356261 -66373861343834643161343665616139373466346130353233623135656663373230333630306132 -66653233623632303463343261653764636230333038623936353138356565623061643432346265 -36326263613833386237333566333534636238336539303638643233376331616331626636666635 -66303535383838336135306439306239323531343331343832376636363931626663333337616464 -61656532623437363634343039313534356565383361303562336666383561333235303939303339 -65326336633631313133396263326139636535346166373333343934316435363435353236363331 -30396231383336326461326462393633353739346463653636393331313534613466363738376263 -62663264643833333032343839373165353262663637376635373430333631316532336335653038 -32303835333461363635306664653331613164316264613632326131633639666263633234306336 -30656562383439333639623534303164323231626337396362373235383530323731626139333335 -38386238396565643533303030393364373735376564373765373632386335613432313735336132 -63306231373038366131353934393735323932633233646439383666366333653535373634386333 -31643130376266626236353937633235623765323462396130383831376366313337643939363366 -62326439623135393539366133386337353964306637343238633730363639373831633662653565 -64333134336666643465633565333765613835373765663664653331353935666437633566386165 -61656239653264646530306165396234626365326238616566303831353935626362366338613339 -30303364356137323935626662636663383761383935343531666461656537643333346637666365 -37663836386466303433313339663531373432643732333461613739636233336566356139323934 -36663965626330333436373764613730393365616166653866306336633939393765633564626331 -31353131663038323235396564623234396138386237663030316530353337373934323232633433 -39363931656466313639346265363466646631363032353363383662306436346162363431353833 -36303934323364363465343236633064336239643565393034353934373239376139386535373061 -34643365393866636430356338626438386136646161376538363762336265653632643334336331 -38646131386463666135376162613864636534366264336630356135316637393135646630333733 -64343731633836306565623238313936656164653038393236356130306162346331396231393436 -38303435383963626434323230643832303838646239646263353737323166613965623734313933 -38313164313062373263326138656239366135383361616436333539356331383064363562376366 -31333838313664653432373937306239363631626234336136396364396166656234623365616237 -36363434393537396639633062626639353738386232393066333034343132303831366362623031 -36643862376637643739626662656239633361653933313130646661656332316535396230386162 -39373738366138373130643636663339613732626532383465316365363638386361623838656630 -66636262666536303739616361343763366135313835353938323330363635343135633138306361 -61346433336366643430646334336136346136646166363962613336366239653236373135376138 -61316464646264316532333839626634623165336334323836643130323137303632353232616631 -32653036313133333237363233653932366334623133646565343461373132306365313331373335 -33623561623163383562366137663733343833333136353738386535313439386164346164383565 -32393331373130633065323266623465613732353431623234653435383133373537363436346463 -63323666666136613034643237623463383462326334343334303731363961303438646330646562 -64336132663561353565376361306331363138306638353834613231323331356238343730616164 -35666432626436376539373633383466343732616430333661613865653530366439626131303233 -63353633613232636238316532633231633539633363623734316337633764333430366466626430 -34336534616537656531613538356231343061326462333663653662343762666131393765306462 -34353435626337656362663637613763623537623534666336643833643963626265303266623434 -32323836663432313438363530663839653637346562626566316163373137363063633564323731 -64336638616661366437656431396435303439373132383330646635306435646138633531336461 -39663366643961303832323838643530653632383230373366396531646639336138373434396431 -32383064313533393735336235356432646162353236343365313435313666363935333835336331 -39316330306336396637613634306531346531393036326536653961313438623664643362623937 -36363535383562663535343332363064326530636335343963366166333365613234643035343764 -62366334623332653931393836303934333139636635363638303734663230363537663338383461 -34366661313931316639303762373131636530363232386263313361666561313034343033626333 -63386630396136316338653134306632323664396466343361306636623164616562393430353732 -32646537653963383639616530646561653065386631386138616130343936333935393861643161 -36613166633561646138336539373731373939633234633861636634396563646137316134636463 -30363133633131303632316634376361623637373133393234393362386338626536653438666539 -31633837353935346438316133333136326237303430313533643265363966396537373161346236 -33656163636338363565653834636461333837653433633639393361636565613337653766353765 -36613834633631346136366232663539636566666339343939383732306537396465373932646434 -31663736323535353763623633366535333262636234316363366537393531666430633631373364 -36393766343863353135333864656536393739636563363631306362336665616636303832666330 -66623336663566316330333337343064626366323563613463646338613433623637363064656639 -63626465613963333932346131656639653239353034613862323565666436316338666563623836 -37356465633635363834316564313839366137656232346637663231373261393035623035643065 -30316663613732616237376265363561326164313631366466653139323534623531323537623164 -35383365336438356538343934356134646337626635326266336533386262323262323734663534 -35313239393232393435346531643138376362336437323963613933623739343463613831326539 -61643939393461386534656664366361363062643939393962376664353563643635373336333266 -39383132363964656438623031366336306362353736633634353033656539376233343131653030 -62653736663637653166373930313664333930346262396339386432353066646465373937393964 -35656538373064643936323731393436386665353838313563613832323834316539366163373638 -30623365333266393331396135613663393139303136663636383766613731636435386635666132 -30633664303830353137393438303666656235616332393132613364333238646566303363363365 -61636132663366666139363965663239623764366334333432633830316636303263373331313931 -30323337383931353363343231656632663534323435353338646664393635323733623166343830 -39633865336666363635346462306261613935383666396261653531396164383961383830333434 -39303063383365306435303430633834386566316332313864373664383766646463393734343961 -34656339343761613863363831393933396438636332333339393433636435316330393634353032 -66303139333933393164363735303534346332336564333366343032353631396633366337333831 -63616637313262316335386230343034323038613530663432353764656337343638383135316361 -32376339333235316335656638613434306633316631376230383434303234666532303662333630 -32626562656363633837316335643162363232623265396362653264313934336566366130376634 -63633132626539353138373263313135633265623761393063383136376130646132643664343431 -32616136306335336636343434366236376536663730643638636234623136383262643766613137 -34666431333265633063356566623139666266643138653365613961613532343337346161643337 -65316332393762386133633030366534353763323738303437636537326137306365626632383630 -34363566643436376535386134383638313237643934303931653839616637643036373134363830 -66656463303331626430663063356239373434386361323139343838333763343565396637356561 -63386436353934646336616537323233336339326261653466653135653735316463653666343231 -36343261616338303864343162646338323339633634663032306433313138376539643236393462 -35333734306662393836386231306466616133383762353464343965376132643634393237396164 -34373831316634626133353735376661633464373238383561316565616331666431356165623039 -34386130363761663365666238323534393933313866333030643132346563646463393063333764 -64646432616231343032633965643337363234393435346430303931633665623362306465383962 -39383437623761613339353733643862376462623166636264303833666437393231656437663764 -65313065323134623265333466373461306639326465326362646532643037333230373837653230 -62383033363561343735623639626565333666636563306639643139666134383762353261343864 -34386334306237363762313465333163643037626534613830663463653563663537396231356431 -36646139333534643765653732303330343532373933656537633465326632663539373638323865 -33386363333065373565336566613333313738313437613764663664633032393865313331633433 -65336664386437353034366638643533366365393864363033366363626630336132313038653430 -30396539313865646333666462643061666566313033613634326537363132663462623861666137 -66386663633634396163666164316631393635623938333137653361363963303235363732623930 -30376632376264343036643461613438396531373865613162643734303963373539363464343337 -39653066626635626164613435643738616534623064306338303734653830633737386531393435 -61303461666666363230643638626230353839653962663439353166346635376438623832663637 -35663738356136363165653133323666383935363566616135376161356165316239373661643039 -35353666326336623561356635343232393137383536356565353762616639383932313464383063 -35393734306636313561383133393961646132633639363136326332366338633535666139643634 -39303735653564656232326339313238383137333630383539623139356539323561353366323131 -37393130356639623133653134623131633031393633386330653034353166376466633739303638 -66613534393331383438326230313936653532336134643632306634323530343630363236646235 -38656463343932333637613866386232376162623939626466303063306466623132643731623338 -63656662376132653238343537343265373236306134376565323961323264383830363065323738 -33366137323535633732316563386237646230613339666337386162633166353533346565393337 -34316233323734643262663532373964346331393031356134373266616265373564386236616237 -38373730623261346365353363383833633964383132623361343333373637386339313262396633 -37666437613361646337323039343938353466346538356138376130313337633533666137626266 -66313838323638626230306138623135363266346366316339313164626233656237376266353762 -65356461346239613066613038353735626233383939666330393131363064363337356330363435 -39656633616166356430343564366433323864333236623934356335336661346338653834363362 -39343435653934643132363931363334366631646463306261666537303938363633336464666537 -38336632643561633461383965333264376462306131666232623735313265373832343762393434 -61386534393034353363613230626330393234326234363837393738376634633561613562323137 -66613930306330623434323539366333663364396165663466303365653331363431656365353266 -63623430353734616439393735623564626638313336613636383438363531306234663939383066 -37623165333233643465363334663433663034636438613433633966306334376462316233343338 -61333530376236333134306164323263633266376666663030303438646335393661316137646237 -32383865636433366635323134663938383339393933656438633662333334313264643338636563 -64343533643432386164333630366531383434333231393762613136616435376534653530346364 -64396531383363376437613366613066633534616163396133323835323431353034373563306536 -35616335636463623565636534346435643463376330333962353261663062613034653863373834 -37653566383164373263616265643536343037346464633930303935393337336333616338633730 -35316166656164356535613364386366373666306131373063376465663935303530666432383435 -65393934396639333765313933643263306337623635623930656430343361653861653039323861 -62326165313038313137323539343934366134383630363632653939633331626566653663613666 -30336161616136613034353133663738646464306164663931373365343664373337303564643565 -31623132396431336130396236656136656335336236306364656164353431633136343732663631 -37316366633837323961643338343538653933306664356236373165333464643032623864633538 -32613030313563343930653863346638393662303030396537343264326234643735323532376337 -61313936623663323961333364306664613331626233393430626632373765333832616136333065 -65323837373231633439333437616536306466656530646332386164373963386431653532666262 -62386234393432633430636439663331386235366630333630383336363664663333383331386362 -31376362643263396634623662373134343039323663343433383836663261376463306337656236 -32633536336238326337336666313263613433353739333530316630653735636133303635313436 -66383761613932653132663939353734623663333736666462363235333336333733323963623663 -33616265636664373638656363636363656639373634353732366664363565383737313863666139 -35303364343063663764393261333336373864373839666664356166626238363035343163653131 -63316436373331393461626164346362366530636335613966353335376334643433333963396563 -39373364356439663363333566656565616130643037613332363937313964363433613436666230 -6232 +30373030336530356533636231303630303132666434393339663833366639366234313434333635 +3763366235343633303836383563333734643766346336390a383732313137623263343836616531 +35656262386162346434343462396166653435633330613366633263356638373061643833386336 +6438373137326563380a353962396135393365376133626538303862396236363330636564323538 +33616565363066633234633330386338316133613931613931653963356562323336396437313130 +33316331333166393161376635306237363033633535613761373762393761363165623462336466 +63663265376532663262323463643462336136373639316565396436653966336461656164316135 +64303235386236316136623063653133363961303733376166363462303565393761636632666230 +66333263626537343935383765303565626237346235356338373063633133326465653133366362 +62326563666531343166663563393932393938363663393132616561336363643037363735393233 +38336233346138356262616533643835396230663563623237616461626265666339613161616331 +34353263363065646336663538663561373536656364623061633863643137633230643931333662 +34386433326661336362636466386566636561623339383438626530333431313039623037616430 +62333832363531323363646238363733316237646133343337363434373465353463363935323233 +37326335343333636337626337643763323336623238636463623933646663356165363062383835 +30636630303534636436326530313030626565393439643238353433396638363133353865643135 +39666464323537383435636636303930396338333332363264306664383538666236616436353939 +61316564623964613839343533316438393337616363633033636231313937366139373031616236 +61373763303231376437313332343438376132633361353661336232333966313338373262383663 +36363931376132333763363332313534616133326437303637343139383861353138623465623736 +62643062396536633730663130353730356637353533616635343166643636616163343832363866 +65646266383062373635306239323565383039646334346535653962316434393365646461653366 +37396163396362313535393138663833653435353934623432616538663565353165363732316662 +35316332363030363963666136653437316639303766376431333738343061663835306339336337 +64346137313763656531306636333030323162646533633735646435613566616532363066353361 +32326439633432656633303732626361636238323735636631633732383635656436373435343931 +64356437326633326163306430306432326134373366616364656361383133656531653333633164 +37653934643061343237376664643331623633653437336534313261643765303134626536393938 +33393161363539313130376132313864666539363865393035626463313539376635393135633565 +35306435636363613735343136663939663435653135376337643034396634623135393039366265 +36326332386564303233623863363236356535323538616237303262613261363964643839363837 +63336536613139333939343964323835626433373464343730353864303532366535343636663362 +34333935316630323137366130356135643962373532346334333931366139356434316431393234 +61386632353434323432306130386239663931306538323335333231366234383132623936343465 +32393765316136663862663334616234326165336364316137396234343532643639366537343731 +33383938363530323337333838643835653238376532376431393439636266306634386233383937 +64623230353533643362633563663537343732646261353763366363343231376562336565333362 +65623665326136383764373761626461653238303335616463636333383235393037323530636630 +65333036653062386332653433366532383132376362356264373432653564316530656163396161 +34386133326133393638653961383032323830366232346134323663653436383034356334643137 +37663062396534616563363166333535336466373335613733323864363736633263646438336662 +39643333366565383130346430623637666132656330653833393836313566393461656638653938 +32623664363835376366353532646564643235356637363432363164353837643066643932383163 +34643232623734613738613638306664326333346439383830326662653934636536386166323239 +63626264613661336663316636353436643262353733303666663866346431353533353435346461 +34343763316364303061353030626431613434383062363737356131366234626137656331383433 +61653339616336376562326638343462343435613632656632353532376438623462656266326539 +35336465366133386534343466323936663934306363353462356530323531383961633336393234 +37366266343831386462626539306637623165363066616164393635373631393737613830363661 +34656530336634336531373830613039616231343238653637623532386538626365386364353930 +32396535643435306231386636363562623239393461393237303030353361633639626632653562 +31663261653663356266376264623762353261613163303430623738343939336438333532353436 +62343465353033343034373837376538323239653064623135613766343463373436663236616366 +30313062663861646331316362653230653139623937343538346230613832623965366666323365 +35623635646263396436346162333835343335623364393037353366343537336462636163303761 +37346663306339333264643034393630613833343163306430396333656637316662616439343136 +35336366306535333733633465343538383462616631326433666663303638633837313065656666 +34326136363433303264353331393133626639343166303364343065333266646439663463353833 +62346637363265643736616238663033666563326462633562643530383862616265306439376439 +63353432343534303433353138663461303165313433353866303838346338363761376333366138 +34353035356132383836313134643363636532353834323438313933346534333238646532326263 +30663964326564623164306664323265383134616165353634653261383931326663356137363636 +36383663666537373863623532376165343334633031626661306664663139616637636162373838 +32663863393765343431613536386133653731363266663566303665316435363830303035656230 +31366139636332336531326131346237653732313337343336393463373438373531653262363035 +39613739323832353332643030343861373836653064326139343965613563363166363236343733 +39653666333438613131396566623237643765393435313163333031383461303366396439353437 +31376562323231386133356231343366623363396437323866326234613034353664323163646631 +36386436653465306665646130616563646564343764346532363961663762303132336137356231 +61376565333036313033633732616430343266356435376434386266306432633633316361656538 +66393266383735336632653834373561636261663738333039653934636434343165353065353566 +34616431616430306234313032346235383065633734323632643065613634343834386664313336 +33333861306362333032346633636630326562323430323863373532613831313931303164303438 +36353831623262663030313564616634376235383666656434373261643464363264653831386534 +63663931643733313239323138393936373435303263663662613266306162373363353334336430 +62393835363263666136646631653738393432346331353538306666663864323233386665306332 +39353839313731313832333139613830396539306133363236653138643161323337356534643037 +38626636353465636166313965346364663238363032383537306532303332643839346230666564 +66613261396633346365633730633738306438326234386438373234393537353635626664616239 +35363737363537326132376638373839326139643135333234333764626162343766623531643265 +32656464383064626566653332323366306138373537663634333833613932666264306164666435 +39623335323163636135313361376231316231626335343765323261653134636439643561663463 +33376130356362316334386162306333333038316664636464653463313835356461386464653035 +38626230376339663361306161386332656230353737376133306464626466653038653266646139 +62313937383262633938633735393765616365646165396436653434303835666162373164333938 +61643432343237386133316433613030633638633731323936393562376139353033643562633835 +36393837383261663761393763613039623263643266303637613461626462313162613762383535 +33373064646533366136376563383663633331373161646534653330653566303332616262386564 +30313330346463643032363464346537613430633163306365313866373031393965383134323031 +63653238343338613033633336366235623332646239613235643637313666616263313833366362 +66666633393232303639363966306230663731353730333264663732386235653633303336316139 +33646263303164613437343735616362636364376261323364616131383433633864323132353063 +32646530323838633962613834663564326137663466623935323665343132653037386339613961 +61666363323763636361396330663537386630306637663638343565653437623966313738306366 +37633436373434396231396461656634316638393336653964396266363532343864346264323830 +32616461353630353730393639306231363662663034626434316661653936636262393761633934 +33393863383461316532373839663432336666356332393562393634386533346262306331643733 +62373162643763616637316233636163636363363639303662313630336233353162343437356533 +30303339666136343665313038643930383539313035373566336139643766353534386232343330 +32633836356435326332666266393237353162646537376564656236353138386531323032626636 +37373963666666346462343938656466343766323434373064363736386134656232636631653535 +38643739333230313265643132356430393035656662356465623836323038663237383439396137 +33383335356433383030393131666233663832643638646537363665396435323261613732303363 +63653634666164636632306462346166623237303936326435326431636630323562613233386462 +64376562316362653937646266366232653733663764633738303162616538656661326261643139 +30376233633835333637356134626266376566393962353639663039323439616639316465353038 +32373136643162633836663937373664306665643638366237653631313631393934643135393261 +35353338393761383431326631633166653932313335346165653364316431333831306135643339 +63316538343963336237653366343836343330656364373661343866656235303931633432313966 +35393631333263363533316537616132323038303230653838326266336434336532346439626537 +30656334316239356261386633393635643563336230326362303034633235323830343435646136 +63333237346164323763633665636234313263393366636233393538353335383933636331656261 +39313733383238613634363139326536616237353031666232663161653763326162386464626533 +39633931353035356561643564656637633163396561356430663636313231333962363765373536 +62613366616538306331623537613564353437366239616332626237623466613931653339333431 +39646239383438623063663063333861356536326637656337303561346432333539656431373733 +31316461343636313430303434336535353430353932373536366162333938646464653763356237 +39363938373065663732393862383031336438396164366664323130343734323130306662343138 +35383734663465303034393561306264316139623265386134323162373734393364393230333234 +31383336363832373962613535346136353037383065363066653935373435383037313764636266 +65383133333433326662626639356162303261393866373732366665353465356534333431633639 +39303539346266316132643537393564346562353533363438376363613036613739633939353564 +37313764316661663939396664376538626437366534383630613030323465396535366537346263 +31636338353233306333653962373763353439346337323839633238636336653439386362333533 +31653264343435633432646633386531386139366461646133396334326631333632393462666232 +35383735356232336335336435343932613433393530653633616466373965383935383861366136 +31303936616333396638326465633531373164626232356362613130376434303633366238336666 +37643035303932393265363037646430343866323163386234633739393938656530646533633163 +61653135633436653566303832383564663462353235353134313033616364636561643663383835 +30326561653934636164636363363736346630316263616664386233373535356339336239653333 +32336362363033616337663936623331663134656665643739626637353739666231643766346134 +63353631663631643436633935306261373939323530363639666366373531626231366138643766 +34623137643233333232303339323464333566633038333539356337306134383330353632383965 +34626461316261363830376664343336666132376365333635666131653933323562666635353763 +66376335663861623332623436333161353161336337313136613531646632363531383138343130 +39306330336239393965383939333765633532376539653661353965373030313038666161313265 +37653164343832333937356434646230623736646561623561643135626531343063626636663333 +30363937613834663434313038653739613933656534626532326136323336386164366339313830 +35396130386634666363656237373835313563343961633838383766663933376439326364616530 +64383136623739623137363562666431333565643130346166663531373738643761363337333536 +38313866363339393066643339653734316266393037396432356138303137343536313134623434 +62623362383863303539663264386430353436613261643865666534356538626538623630643766 +66646266336238396230613838626161336637313564303465663034653232306363306430633438 +35303364356531636565303965323539393233663566646562383863323461396166623032396338 +36346537343633643864323163623464623539616434376137313538646333666339626437656330 +31326463666335626361616566383065363264356663323435646337663566636436356336376531 +38636466386539663937326462636464613638653833666263333134396236313432643030366138 +66326364623435356633666463333066373238363864356230633634353764616331303163636266 +36306239323561326134393665643031663362636535656139376237383130616663316362326366 +61623236356562346233643332663261626333646638373262613664623935626266306230356164 +32336363383538333935333062386366653839303139366565343933346266623637363534393335 +31346431313631373535306138633930383536613636393034343434623664376237373066633036 +30323864613430333239326366613439613632633933656631356237333930303330326264646563 +38303066323563303262663530326637663964373039336431353339626436363335326466313262 +35386533633833363038636339663265626261373464633565666166633135333461363165353931 +63633332613966663133 diff --git a/ansible/infra/403_service_probe_registration.yml b/ansible/infra/403_service_probe_registration.yml new file mode 100644 index 0000000..a8c8b35 --- /dev/null +++ b/ansible/infra/403_service_probe_registration.yml @@ -0,0 +1,47 @@ +--- +# The per-service health probes. +# +# These are NOT the same thing as the systemd checks in infra/401. Those answer +# "is the unit running"; these answer "does the service actually respond" - an +# RPC call to bitcoind, a TCP connect to Fulcrum's Electrum port, an HTTP fetch +# from the Mempool backend. A process can be perfectly `active` and useless, +# which is precisely the gap these close. +# +# The checks themselves live in each service's own role, deployed by that +# service's playbook. This play only registers where they report, because the +# endpoints must exist in Gatus before the first push arrives. +# +# They used to push to Uptime Kuma. The scripts now POST with a bearer token +# instead of GETting ?status=up, and each host uses its own token. + +- name: Register the per-service probes with Gatus + hosts: observability + become: yes + + vars: + probes: + - {name: bitcoin-knots, host: knots_box_local} + - {name: datum-gateway, host: knots_box_local} + - {name: fulcrum, host: fulcrum_box_local} + - {name: phoenixd, host: vipy} + - {name: forgejo-runner, host: forgejo_runner_local} + - {name: mempool-mariadb, host: mempool_box_local} + - {name: mempool-backend, host: mempool_box_local} + - {name: mempool-frontend, host: mempool_box_local} + + tasks: + - name: Build the probe endpoint list + ansible.builtin.set_fact: + probe_endpoints: "{{ probe_endpoints | default([]) + [{ + 'name': item.name, + 'group': 'probe', + 'token': gatus_push_tokens[item.host], + 'heartbeat': '16m'}] }}" + loop: "{{ probes }}" + + - name: Register the probe endpoints + ansible.builtin.include_role: + name: gatus_endpoint + vars: + gatus_endpoint_name: probes + gatus_endpoint_external: "{{ probe_endpoints }}" diff --git a/ansible/roles/bitcoin_knots/defaults/main.yml b/ansible/roles/bitcoin_knots/defaults/main.yml index 3a12ad1..1045e77 100644 --- a/ansible/roles/bitcoin_knots/defaults/main.yml +++ b/ansible/roles/bitcoin_knots/defaults/main.yml @@ -51,6 +51,9 @@ bitcoin_group: bitcoin # check, exit honestly, report nowhere. Any endpoint accepting an HTTP ping # works; nothing here is specific to a monitoring product. healthcheck_push_url: "" +# Bearer token for the Gatus external endpoint. Required whenever a push URL +# is set: Gatus rejects an unauthenticated push with 401. +healthcheck_push_token: "" # --- Logging ---------------------------------------------------------------- # The live node logs to a file. Set to "" to use printtoconsole=1 (journald). diff --git a/ansible/roles/bitcoin_knots/tasks/healthcheck.yml b/ansible/roles/bitcoin_knots/tasks/healthcheck.yml index 34b5529..c6c8db3 100644 --- a/ansible/roles/bitcoin_knots/tasks/healthcheck.yml +++ b/ansible/roles/bitcoin_knots/tasks/healthcheck.yml @@ -24,7 +24,7 @@ dest: /etc/systemd/system/bitcoin-knots-healthcheck.service owner: root group: root - mode: '0644' + mode: "0600" - name: Create systemd timer for Bitcoin Knots health check ansible.builtin.template: diff --git a/ansible/roles/bitcoin_knots/templates/healthcheck.service.j2 b/ansible/roles/bitcoin_knots/templates/healthcheck.service.j2 index a2ba83d..9df056e 100644 --- a/ansible/roles/bitcoin_knots/templates/healthcheck.service.j2 +++ b/ansible/roles/bitcoin_knots/templates/healthcheck.service.j2 @@ -7,6 +7,7 @@ Type=oneshot User=root ExecStart=/usr/local/bin/bitcoin-knots-healthcheck-push.sh Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} +Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} StandardOutput=journal StandardError=journal diff --git a/ansible/roles/bitcoin_knots/templates/healthcheck.sh.j2 b/ansible/roles/bitcoin_knots/templates/healthcheck.sh.j2 index 4e6ea9d..538b8dd 100644 --- a/ansible/roles/bitcoin_knots/templates/healthcheck.sh.j2 +++ b/ansible/roles/bitcoin_knots/templates/healthcheck.sh.j2 @@ -12,6 +12,7 @@ RPC_PORT={{ bitcoin_rpc_port }} RPC_USER="{{ bitcoin_rpc_user }}" RPC_PASSWORD="{{ bitcoin_rpc_password }}" PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" +PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" # Check if bitcoind RPC is responding check_bitcoind() { @@ -46,8 +47,14 @@ report() { # URL encode spaces in message local encoded_msg="${msg// /%20}" - if ! curl -s --max-time 10 --retry 2 -o /dev/null \ - "${PUSH_URL}?status=${status}&msg=${encoded_msg}&ping="; then + # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's + # GET with ?status=up. The callers still pass up/down, so the mapping is + # done here rather than at every call site. + local _ok=false + [ "${status}" = "up" ] && _ok=true + if ! curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" \ + "${PUSH_URL}?success=${_ok}&error=${encoded_msg}"; then return 1 fi } diff --git a/ansible/roles/datum_gateway/defaults/main.yml b/ansible/roles/datum_gateway/defaults/main.yml index b9829f9..0c30aa7 100644 --- a/ansible/roles/datum_gateway/defaults/main.yml +++ b/ansible/roles/datum_gateway/defaults/main.yml @@ -56,3 +56,6 @@ datum_pooled_mining_only: true # WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means # check, exit honestly, report nowhere. healthcheck_push_url: "" +# Bearer token for the Gatus external endpoint. Required whenever a push URL +# is set: Gatus rejects an unauthenticated push with 401. +healthcheck_push_token: "" diff --git a/ansible/roles/datum_gateway/tasks/healthcheck.yml b/ansible/roles/datum_gateway/tasks/healthcheck.yml index 211b85b..b5ccb90 100644 --- a/ansible/roles/datum_gateway/tasks/healthcheck.yml +++ b/ansible/roles/datum_gateway/tasks/healthcheck.yml @@ -19,7 +19,7 @@ dest: /etc/systemd/system/datum-gateway-healthcheck.service owner: root group: root - mode: '0644' + mode: "0600" notify: Restart datum-gateway health check timer - name: Create datum-gateway health check systemd timer diff --git a/ansible/roles/datum_gateway/templates/healthcheck.service.j2 b/ansible/roles/datum_gateway/templates/healthcheck.service.j2 index 5dcd3f5..e21672e 100644 --- a/ansible/roles/datum_gateway/templates/healthcheck.service.j2 +++ b/ansible/roles/datum_gateway/templates/healthcheck.service.j2 @@ -7,6 +7,7 @@ Type=oneshot User=root ExecStart=/usr/local/bin/datum-gateway-healthcheck-push.sh Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} +Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} StandardOutput=journal StandardError=journal diff --git a/ansible/roles/datum_gateway/templates/healthcheck.sh.j2 b/ansible/roles/datum_gateway/templates/healthcheck.sh.j2 index ba8d069..43ec6b2 100644 --- a/ansible/roles/datum_gateway/templates/healthcheck.sh.j2 +++ b/ansible/roles/datum_gateway/templates/healthcheck.sh.j2 @@ -5,6 +5,7 @@ # systemctl is-failed datum-gateway-healthcheck.service # Reporting anywhere else is optional and generic. PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" +PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" STRATUM_PORT={{ datum_gateway_stratum_port }} check_datum() { @@ -19,8 +20,14 @@ report() { # No push URL is normal, not an error: the exit code below is still a # complete answer for anything reading unit state. [ -n "$PUSH_URL" ] || return 0 - curl -s --max-time 10 --retry 2 -o /dev/null \ - "${PUSH_URL}?status=${status}&msg=${msg// /%20}&ping=" || true + # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's + # GET with ?status=up. The callers still pass up/down, so the mapping is + # done here rather than at every call site. + local _ok=false + [ "${status}" = "up" ] && _ok=true + curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" \ + "${PUSH_URL}?success=${_ok}&error=${msg// /%20}" || true } if check_datum; then diff --git a/ansible/roles/forgejo_runner/defaults/main.yml b/ansible/roles/forgejo_runner/defaults/main.yml index 4a80d73..aba6b61 100644 --- a/ansible/roles/forgejo_runner/defaults/main.yml +++ b/ansible/roles/forgejo_runner/defaults/main.yml @@ -36,3 +36,6 @@ healthcheck_service_name: forgejo-runner-healthcheck # A pull-based monitor (Prometheus node_exporter textfile, say) needs this left # empty — it reads the systemd unit state instead. healthcheck_push_url: "" +# Bearer token for the Gatus external endpoint. Required whenever a push URL +# is set: Gatus rejects an unauthenticated push with 401. +healthcheck_push_token: "" diff --git a/ansible/roles/forgejo_runner/tasks/healthcheck.yml b/ansible/roles/forgejo_runner/tasks/healthcheck.yml index 0abdf1f..42fbaf7 100644 --- a/ansible/roles/forgejo_runner/tasks/healthcheck.yml +++ b/ansible/roles/forgejo_runner/tasks/healthcheck.yml @@ -27,7 +27,7 @@ dest: "/etc/systemd/system/{{ healthcheck_service_name }}.service" owner: root group: root - mode: '0644' + mode: "0600" - name: Create healthcheck systemd timer ansible.builtin.template: diff --git a/ansible/roles/forgejo_runner/templates/healthcheck.service.j2 b/ansible/roles/forgejo_runner/templates/healthcheck.service.j2 index aae9eb5..173f42b 100644 --- a/ansible/roles/forgejo_runner/templates/healthcheck.service.j2 +++ b/ansible/roles/forgejo_runner/templates/healthcheck.service.j2 @@ -5,6 +5,8 @@ After=network.target [Service] Type=oneshot ExecStart={{ healthcheck_script_path }} +Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} +Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} User=root StandardOutput=journal StandardError=journal diff --git a/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 b/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 index b9d43c3..9aa5ac1 100644 --- a/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 +++ b/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 @@ -10,34 +10,38 @@ # it. Nothing here knows or cares which monitoring product is on the other end. LOG_FILE="{{ healthcheck_log_file }}" -PUSH_URL="{{ healthcheck_push_url }}" +# Read from the environment rather than templated in, so the unit file is the +# only place the token lives and the script is not secret. +PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" +PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" log_message() { echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE" } +# Gatus external endpoint: a POST with a bearer token and success=true|false. +# +# This used to report ONLY success - it exited before pushing when the runner +# was down - so a failure was invisible until the heartbeat window expired. +# Reporting the failure is the whole point of having a check. +report() { + local ok="$1" msg="$2" + [ -n "$PUSH_URL" ] || return 0 + curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" \ + "${PUSH_URL}?success=${ok}&error=${msg// /%20}" || true +} + main() { if ! systemctl is-active --quiet forgejo-runner; then log_message "ERROR: forgejo-runner is not active" + report false "forgejo-runner is not active" exit 1 fi - if [ -z "$PUSH_URL" ]; then - # Healthy, and nothing to report to. Not an error: the exit code below - # is still a complete answer for anything reading unit state. - log_message "forgejo-runner is active (no push URL configured)" - exit 0 - fi - - log_message "forgejo-runner is active, sending ping" - response=$(curl -s -w "\n%{http_code}" "$PUSH_URL?status=up&msg=forgejo-runner%20is%20active" 2>&1) - http_code=$(echo "$response" | tail -n1) - if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then - log_message "Ping sent successfully (HTTP $http_code)" - else - log_message "ERROR: Failed to send ping (HTTP $http_code)" - exit 1 - fi + log_message "forgejo-runner is active" + report true "active" + exit 0 } main diff --git a/ansible/roles/fulcrum/defaults/main.yml b/ansible/roles/fulcrum/defaults/main.yml index cc778d4..3e97fa0 100644 --- a/ansible/roles/fulcrum/defaults/main.yml +++ b/ansible/roles/fulcrum/defaults/main.yml @@ -71,6 +71,9 @@ fulcrum_group: fulcrum # check, exit honestly, report nowhere. Any endpoint accepting an HTTP ping # works; nothing here is specific to a monitoring product. healthcheck_push_url: "" +# Bearer token for the Gatus external endpoint. Required whenever a push URL +# is set: Gatus rejects an unauthenticated push with 401. +healthcheck_push_token: "" # Explicit db_mem in MB. When set it wins over fulcrum_db_mem_percent; empty # means compute from RAM. Set here because the live host had been hand-tuned to diff --git a/ansible/roles/fulcrum/tasks/healthcheck.yml b/ansible/roles/fulcrum/tasks/healthcheck.yml index e86726e..983bffb 100644 --- a/ansible/roles/fulcrum/tasks/healthcheck.yml +++ b/ansible/roles/fulcrum/tasks/healthcheck.yml @@ -19,7 +19,7 @@ dest: /etc/systemd/system/fulcrum-healthcheck.service owner: root group: root - mode: '0644' + mode: "0600" - name: Create systemd timer for Fulcrum health check ansible.builtin.template: diff --git a/ansible/roles/fulcrum/templates/healthcheck.service.j2 b/ansible/roles/fulcrum/templates/healthcheck.service.j2 index 27995f4..b519808 100644 --- a/ansible/roles/fulcrum/templates/healthcheck.service.j2 +++ b/ansible/roles/fulcrum/templates/healthcheck.service.j2 @@ -7,6 +7,7 @@ Type=oneshot User=root ExecStart=/usr/local/bin/fulcrum-healthcheck-push.sh Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} +Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} StandardOutput=journal StandardError=journal diff --git a/ansible/roles/fulcrum/templates/healthcheck.sh.j2 b/ansible/roles/fulcrum/templates/healthcheck.sh.j2 index ee1c7da..4449fed 100644 --- a/ansible/roles/fulcrum/templates/healthcheck.sh.j2 +++ b/ansible/roles/fulcrum/templates/healthcheck.sh.j2 @@ -10,6 +10,7 @@ FULCRUM_HOST="{{ fulcrum_tcp_bind }}" FULCRUM_PORT={{ fulcrum_tcp_port }} PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" +PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" check_fulcrum() { timeout 5 bash -c "echo > /dev/tcp/${FULCRUM_HOST}/${FULCRUM_PORT}" 2>/dev/null @@ -20,8 +21,14 @@ report() { # No push URL is normal, not an error: the exit code below is still a # complete answer for anything reading unit state. [ -n "$PUSH_URL" ] || return 0 - curl -s --max-time 10 --retry 2 -o /dev/null \ - "${PUSH_URL}?status=${status}&msg=${msg// /%20}&ping=" || true + # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's + # GET with ?status=up. The callers still pass up/down, so the mapping is + # done here rather than at every call site. + local _ok=false + [ "${status}" = "up" ] && _ok=true + curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" \ + "${PUSH_URL}?success=${_ok}&error=${msg// /%20}" || true } if check_fulcrum; then diff --git a/ansible/roles/mempool/defaults/main.yml b/ansible/roles/mempool/defaults/main.yml index 2f983c6..9ce8f6a 100644 --- a/ansible/roles/mempool/defaults/main.yml +++ b/ansible/roles/mempool/defaults/main.yml @@ -45,6 +45,8 @@ mariadb_user: "mempool" # push_url is where to report, and is the single plug-in point for whatever # monitoring exists. Empty means check, exit honestly, report nowhere. # The URLs are credentials, so callers pass them from the vault. +healthcheck_push_token: "" + mempool_healthchecks: - name: mariadb label: MariaDB diff --git a/ansible/roles/mempool/tasks/healthcheck.yml b/ansible/roles/mempool/tasks/healthcheck.yml index fa53a9c..1e1ac28 100644 --- a/ansible/roles/mempool/tasks/healthcheck.yml +++ b/ansible/roles/mempool/tasks/healthcheck.yml @@ -22,7 +22,7 @@ dest: "/etc/systemd/system/mempool-{{ hc.name }}-healthcheck.service" owner: root group: root - mode: '0644' + mode: "0600" loop: "{{ mempool_healthchecks }}" loop_control: loop_var: hc diff --git a/ansible/roles/mempool/templates/healthcheck-backend.sh.j2 b/ansible/roles/mempool/templates/healthcheck-backend.sh.j2 index 3a6630a..fe0f57c 100644 --- a/ansible/roles/mempool/templates/healthcheck-backend.sh.j2 +++ b/ansible/roles/mempool/templates/healthcheck-backend.sh.j2 @@ -2,6 +2,7 @@ # Mempool backend health check — managed by Ansible (roles/mempool) # The exit code is the answer; systemd keeps it. Reporting is optional. PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" +PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" BACKEND_PORT="{{ mempool_backend_port }}" check() { @@ -10,8 +11,14 @@ check() { report() { [ -n "$PUSH_URL" ] || return 0 - curl -s --max-time 10 --retry 2 -o /dev/null \ - "${PUSH_URL}?status=$1&msg=${2// /%20}&ping=" || true + # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's + # GET with ?status=up. The callers still pass up/down, so the mapping is + # done here rather than at every call site. + local _ok=false + [ "$1" = "up" ] && _ok=true + curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" \ + "${PUSH_URL}?success=${_ok}&error=${2// /%20}" || true } if check; then report up "OK"; exit 0 diff --git a/ansible/roles/mempool/templates/healthcheck-frontend.sh.j2 b/ansible/roles/mempool/templates/healthcheck-frontend.sh.j2 index b2541d3..8452202 100644 --- a/ansible/roles/mempool/templates/healthcheck-frontend.sh.j2 +++ b/ansible/roles/mempool/templates/healthcheck-frontend.sh.j2 @@ -2,6 +2,7 @@ # Mempool frontend health check — managed by Ansible (roles/mempool) # The exit code is the answer; systemd keeps it. Reporting is optional. PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" +PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" FRONTEND_PORT="{{ mempool_frontend_port }}" check() { @@ -10,8 +11,14 @@ check() { report() { [ -n "$PUSH_URL" ] || return 0 - curl -s --max-time 10 --retry 2 -o /dev/null \ - "${PUSH_URL}?status=$1&msg=${2// /%20}&ping=" || true + # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's + # GET with ?status=up. The callers still pass up/down, so the mapping is + # done here rather than at every call site. + local _ok=false + [ "$1" = "up" ] && _ok=true + curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" \ + "${PUSH_URL}?success=${_ok}&error=${2// /%20}" || true } if check; then report up "OK"; exit 0 diff --git a/ansible/roles/mempool/templates/healthcheck-mariadb.sh.j2 b/ansible/roles/mempool/templates/healthcheck-mariadb.sh.j2 index cbc36f5..922adab 100644 --- a/ansible/roles/mempool/templates/healthcheck-mariadb.sh.j2 +++ b/ansible/roles/mempool/templates/healthcheck-mariadb.sh.j2 @@ -2,6 +2,7 @@ # Mempool MariaDB health check — managed by Ansible (roles/mempool) # The exit code is the answer; systemd keeps it. Reporting is optional. PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" +PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" check() { {% raw %} @@ -13,8 +14,14 @@ report() { # No push URL is normal, not an error. The previous version logged # "ERROR: UPTIME_KUMA_PUSH_URL not set" on every fire, once a minute. [ -n "$PUSH_URL" ] || return 0 - curl -s --max-time 10 --retry 2 -o /dev/null \ - "${PUSH_URL}?status=$1&msg=${2// /%20}&ping=" || true + # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's + # GET with ?status=up. The callers still pass up/down, so the mapping is + # done here rather than at every call site. + local _ok=false + [ "$1" = "up" ] && _ok=true + curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" \ + "${PUSH_URL}?success=${_ok}&error=${2// /%20}" || true } if check; then report up "OK"; exit 0 diff --git a/ansible/roles/mempool/templates/healthcheck.service.j2 b/ansible/roles/mempool/templates/healthcheck.service.j2 index 44548bd..76fa4e9 100644 --- a/ansible/roles/mempool/templates/healthcheck.service.j2 +++ b/ansible/roles/mempool/templates/healthcheck.service.j2 @@ -7,6 +7,7 @@ Type=oneshot User=root ExecStart=/usr/local/bin/mempool-{{ hc.name }}-healthcheck-push.sh Environment=HEALTHCHECK_PUSH_URL={{ hc.push_url }} +Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} StandardOutput=journal StandardError=journal diff --git a/ansible/roles/phoenixd/defaults/main.yml b/ansible/roles/phoenixd/defaults/main.yml index 9fea220..2fcf1ea 100644 --- a/ansible/roles/phoenixd/defaults/main.yml +++ b/ansible/roles/phoenixd/defaults/main.yml @@ -51,3 +51,6 @@ phoenixd_healthcheck_service_name: phoenixd-healthcheck # Empty means check, log, exit honestly, report nowhere. Any endpoint that # accepts an HTTP ping works; nothing here is specific to a monitoring product. healthcheck_push_url: "" +# Bearer token for the Gatus external endpoint. Required whenever a push URL +# is set: Gatus rejects an unauthenticated push with 401. +healthcheck_push_token: "" diff --git a/ansible/roles/phoenixd/tasks/healthcheck.yml b/ansible/roles/phoenixd/tasks/healthcheck.yml index ad5cc22..d439047 100644 --- a/ansible/roles/phoenixd/tasks/healthcheck.yml +++ b/ansible/roles/phoenixd/tasks/healthcheck.yml @@ -19,7 +19,7 @@ dest: "/etc/systemd/system/{{ phoenixd_healthcheck_service_name }}.service" owner: root group: root - mode: "0644" + mode: "0600" notify: Restart phoenixd health check timer - name: Create phoenixd health check systemd timer diff --git a/ansible/roles/phoenixd/templates/healthcheck.service.j2 b/ansible/roles/phoenixd/templates/healthcheck.service.j2 index 4060dbc..6ff21ab 100644 --- a/ansible/roles/phoenixd/templates/healthcheck.service.j2 +++ b/ansible/roles/phoenixd/templates/healthcheck.service.j2 @@ -7,6 +7,7 @@ Type=oneshot User=root ExecStart={{ phoenixd_healthcheck_script_path }} Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} +Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} StandardOutput=journal StandardError=journal diff --git a/ansible/roles/phoenixd/templates/healthcheck.sh.j2 b/ansible/roles/phoenixd/templates/healthcheck.sh.j2 index d2cec21..a1717c5 100644 --- a/ansible/roles/phoenixd/templates/healthcheck.sh.j2 +++ b/ansible/roles/phoenixd/templates/healthcheck.sh.j2 @@ -6,6 +6,7 @@ # systemctl is-failed {{ phoenixd_healthcheck_service_name }}.service # That is a complete answer on its own. Reporting anywhere else is optional. PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" +PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" export PHOENIX_DATADIR="{{ phoenixd_data_dir }}" check_phoenixd() { @@ -25,8 +26,14 @@ report() { # answers the question. The previous version logged ERROR here on every # single fire, once a minute, which is noise that trains you to ignore it. [ -n "$PUSH_URL" ] || return 0 - curl -s --max-time 10 --retry 2 -o /dev/null \ - "${PUSH_URL}?status=${status}&msg=${msg// /%20}&ping=" || true + # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's + # GET with ?status=up. The callers still pass up/down, so the mapping is + # done here rather than at every call site. + local _ok=false + [ "${status}" = "up" ] && _ok=true + curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ + -H "Authorization: Bearer ${PUSH_TOKEN}" \ + "${PUSH_URL}?success=${_ok}&error=${msg// /%20}" || true } if check_phoenixd; then diff --git a/ansible/services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml b/ansible/services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml index 0ca1e13..d67006b 100644 --- a/ansible/services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml +++ b/ansible/services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml @@ -12,7 +12,8 @@ vars: # Preserves the push URL this check has been reporting to. The role knows # nothing about Uptime Kuma — this is just "a URL that accepts a ping". - healthcheck_push_url: "{{ healthcheck_push_urls.bitcoin_knots | default('') }}" + healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_bitcoin-knots/external" + healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" roles: - bitcoin_knots diff --git a/ansible/services/datum-gateway/deploy_datum_gateway_playbook.yml b/ansible/services/datum-gateway/deploy_datum_gateway_playbook.yml index 84de178..2e88a27 100644 --- a/ansible/services/datum-gateway/deploy_datum_gateway_playbook.yml +++ b/ansible/services/datum-gateway/deploy_datum_gateway_playbook.yml @@ -13,7 +13,8 @@ vars: # Preserves the push URL this check reports to. The role knows nothing about # Uptime Kuma — this is just "a URL that accepts a ping". - healthcheck_push_url: "{{ healthcheck_push_urls.datum_gateway | default('') }}" + healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_datum-gateway/external" + healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" roles: - datum_gateway diff --git a/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml b/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml index 031f081..04081ff 100644 --- a/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml +++ b/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml @@ -7,6 +7,7 @@ # move to a role changes no behaviour. The role itself knows nothing about # Uptime Kuma — this is just "a URL that accepts a ping", and whatever # replaces it sets the same variable. - healthcheck_push_url: "{{ healthcheck_push_urls.forgejo_runner | default('') }}" + healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_forgejo-runner/external" + healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" roles: - forgejo_runner diff --git a/ansible/services/forgejo/deploy_forgejo_playbook.yml b/ansible/services/forgejo/deploy_forgejo_playbook.yml index 04d7041..f908586 100644 --- a/ansible/services/forgejo/deploy_forgejo_playbook.yml +++ b/ansible/services/forgejo/deploy_forgejo_playbook.yml @@ -6,7 +6,6 @@ vars: forgejo_subdomain: "{{ subdomains.forgejo }}" forgejo_domain: "{{ forgejo_subdomain }}.{{ root_domain }}" - uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Ensure required packages are installed @@ -91,123 +90,3 @@ caddy_site_name: forgejo caddy_site_domain: "{{ forgejo_domain }}" caddy_site_upstream: "localhost:{{ forgejo_port }}" - - # ═════════════════════════════════════════════════════════════════════════ - # DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. - # - # Every task below is inert: uptime_kuma_enabled is false in - # group_vars/all/main.yml, so they all skip and the deployment above still - # runs normally. Kept because the health-check logic is the durable part — - # when a replacement exists, rewire the push transport and flip the flag. - # - # What was being monitored: archive/uptime_kuma/MONITORS.md - # ═════════════════════════════════════════════════════════════════════════ - - name: Create Uptime Kuma monitor setup script for Forgejo - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/setup_forgejo_monitor.py - content: | - #!/usr/bin/env python3 - import sys - import yaml - from uptime_kuma_api import UptimeKumaApi, MonitorType - - try: - with open('/tmp/ansible_config.yml', 'r') as f: - config = yaml.safe_load(f) - - url = config['uptime_kuma_url'] - username = config['username'] - password = config['password'] - monitor_url = config['monitor_url'] - monitor_name = config['monitor_name'] - - api = UptimeKumaApi(url, timeout=30) - api.login(username, password) - - # Get all monitors - monitors = api.get_monitors() - - # Find or create "services" group - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - if not group: - group_result = api.add_monitor(type='group', name='services') - # Refresh to get the group with id - monitors = api.get_monitors() - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - - # Check if monitor already exists - existing_monitor = None - for monitor in monitors: - if monitor.get('name') == monitor_name: - existing_monitor = monitor - break - - # Get ntfy notification ID - notifications = api.get_notifications() - ntfy_notification_id = None - for notif in notifications: - if notif.get('type') == 'ntfy': - ntfy_notification_id = notif.get('id') - break - - if existing_monitor: - print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") - print("Skipping - monitor already configured") - else: - print(f"Creating monitor '{monitor_name}'...") - api.add_monitor( - type=MonitorType.HTTP, - name=monitor_name, - url=monitor_url, - parent=group['id'], - interval=60, - maxretries=3, - retryInterval=60, - notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} - ) - - api.disconnect() - print("SUCCESS") - - except Exception as e: - print(f"ERROR: {str(e)}", file=sys.stderr) - sys.exit(1) - mode: '0755' - - - name: Create temporary config for monitor setup - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/ansible_config.yml - content: | - uptime_kuma_url: "{{ uptime_kuma_api_url }}" - username: "{{ uptime_kuma_username }}" - password: "{{ uptime_kuma_password }}" - monitor_url: "https://{{ forgejo_domain }}/api/healthz" - monitor_name: "Forgejo" - mode: '0644' - - - name: Run Uptime Kuma monitor setup - when: uptime_kuma_enabled | default(false) - command: python3 /tmp/setup_forgejo_monitor.py - delegate_to: localhost - become: no - register: monitor_setup - changed_when: "'SUCCESS' in monitor_setup.stdout" - ignore_errors: yes - - - name: Clean up temporary files - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - file: - path: "{{ item }}" - state: absent - loop: - - /tmp/setup_forgejo_monitor.py - - /tmp/ansible_config.yml - diff --git a/ansible/services/fulcrum/deploy_fulcrum_playbook.yml b/ansible/services/fulcrum/deploy_fulcrum_playbook.yml index 8927aba..1b7b963 100644 --- a/ansible/services/fulcrum/deploy_fulcrum_playbook.yml +++ b/ansible/services/fulcrum/deploy_fulcrum_playbook.yml @@ -8,7 +8,8 @@ vars: # Preserves the push URL this check has been configured with. The role knows # nothing about Uptime Kuma — this is just "a URL that accepts a ping". - healthcheck_push_url: "{{ healthcheck_push_urls.fulcrum | default('') }}" + healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_fulcrum/external" + healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" roles: - fulcrum diff --git a/ansible/services/headscale/deploy_headscale_playbook.yml b/ansible/services/headscale/deploy_headscale_playbook.yml index 11c967e..e181f24 100644 --- a/ansible/services/headscale/deploy_headscale_playbook.yml +++ b/ansible/services/headscale/deploy_headscale_playbook.yml @@ -7,7 +7,6 @@ headscale_subdomain: "{{ subdomains.headscale }}" headscale_domain: "{{ headscale_subdomain }}.{{ root_domain }}" headscale_base_domain: "tailnet.{{ root_domain }}" - uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Install required packages @@ -254,125 +253,6 @@ # All API operations require a valid Bearer token in the Authorization header reverse_proxy * http://localhost:{{ headscale_port }} - # ═════════════════════════════════════════════════════════════════════════ - # DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. - # - # Every task below is inert: uptime_kuma_enabled is false in - # group_vars/all/main.yml, so they all skip and the deployment above still - # runs normally. Kept because the health-check logic is the durable part — - # when a replacement exists, rewire the push transport and flip the flag. - # - # What was being monitored: archive/uptime_kuma/MONITORS.md - # ═════════════════════════════════════════════════════════════════════════ - - name: Create Uptime Kuma monitor setup script for Headscale - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/setup_headscale_monitor.py - content: | - #!/usr/bin/env python3 - import sys - import yaml - from uptime_kuma_api import UptimeKumaApi, MonitorType - - try: - with open('/tmp/ansible_config.yml', 'r') as f: - config = yaml.safe_load(f) - - url = config['uptime_kuma_url'] - username = config['username'] - password = config['password'] - monitor_url = config['monitor_url'] - monitor_name = config['monitor_name'] - - api = UptimeKumaApi(url, timeout=30) - api.login(username, password) - - # Get all monitors - monitors = api.get_monitors() - - # Find or create "services" group - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - if not group: - group_result = api.add_monitor(type='group', name='services') - # Refresh to get the group with id - monitors = api.get_monitors() - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - - # Check if monitor already exists - existing_monitor = None - for monitor in monitors: - if monitor.get('name') == monitor_name: - existing_monitor = monitor - break - - # Get ntfy notification ID - notifications = api.get_notifications() - ntfy_notification_id = None - for notif in notifications: - if notif.get('type') == 'ntfy': - ntfy_notification_id = notif.get('id') - break - - if existing_monitor: - print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") - print("Skipping - monitor already configured") - else: - print(f"Creating monitor '{monitor_name}'...") - api.add_monitor( - type=MonitorType.HTTP, - name=monitor_name, - url=monitor_url, - parent=group['id'], - interval=60, - maxretries=3, - retryInterval=60, - notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} - ) - - api.disconnect() - print("SUCCESS") - - except Exception as e: - print(f"ERROR: {str(e)}", file=sys.stderr) - sys.exit(1) - mode: '0755' - - - name: Create temporary config for monitor setup - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/ansible_config.yml - content: | - uptime_kuma_url: "{{ uptime_kuma_api_url }}" - username: "{{ uptime_kuma_username }}" - password: "{{ uptime_kuma_password }}" - monitor_url: "https://{{ headscale_domain }}/health" - monitor_name: "Headscale" - mode: '0644' - - - name: Run Uptime Kuma monitor setup - when: uptime_kuma_enabled | default(false) - command: python3 /tmp/setup_headscale_monitor.py - delegate_to: localhost - become: no - register: monitor_setup - changed_when: "'SUCCESS' in monitor_setup.stdout" - ignore_errors: yes - - - name: Clean up temporary files - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - file: - path: "{{ item }}" - state: absent - loop: - - /tmp/setup_headscale_monitor.py - - /tmp/ansible_config.yml - handlers: - name: Restart headscale become: yes diff --git a/ansible/services/lnbits/deploy_lnbits_playbook.yml b/ansible/services/lnbits/deploy_lnbits_playbook.yml index 5d0c21d..7b89f33 100644 --- a/ansible/services/lnbits/deploy_lnbits_playbook.yml +++ b/ansible/services/lnbits/deploy_lnbits_playbook.yml @@ -6,7 +6,6 @@ vars: lnbits_subdomain: "{{ subdomains.lnbits }}" lnbits_domain: "{{ lnbits_subdomain }}.{{ root_domain }}" - uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Create lnbits directory @@ -152,123 +151,3 @@ caddy_site_upstream: "localhost:{{ lnbits_port }}" caddy_site_headers_up: X-Forwarded-Host: "{{ lnbits_domain }}" - - # ═════════════════════════════════════════════════════════════════════════ - # DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. - # - # Every task below is inert: uptime_kuma_enabled is false in - # group_vars/all/main.yml, so they all skip and the deployment above still - # runs normally. Kept because the health-check logic is the durable part — - # when a replacement exists, rewire the push transport and flip the flag. - # - # What was being monitored: archive/uptime_kuma/MONITORS.md - # ═════════════════════════════════════════════════════════════════════════ - - name: Create Uptime Kuma monitor setup script for LNBits - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/setup_lnbits_monitor.py - content: | - #!/usr/bin/env python3 - import sys - import yaml - from uptime_kuma_api import UptimeKumaApi, MonitorType - - try: - with open('/tmp/ansible_config.yml', 'r') as f: - config = yaml.safe_load(f) - - url = config['uptime_kuma_url'] - username = config['username'] - password = config['password'] - monitor_url = config['monitor_url'] - monitor_name = config['monitor_name'] - - api = UptimeKumaApi(url, timeout=30) - api.login(username, password) - - # Get all monitors - monitors = api.get_monitors() - - # Find or create "services" group - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - if not group: - group_result = api.add_monitor(type='group', name='services') - # Refresh to get the group with id - monitors = api.get_monitors() - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - - # Check if monitor already exists - existing_monitor = None - for monitor in monitors: - if monitor.get('name') == monitor_name: - existing_monitor = monitor - break - - # Get ntfy notification ID - notifications = api.get_notifications() - ntfy_notification_id = None - for notif in notifications: - if notif.get('type') == 'ntfy': - ntfy_notification_id = notif.get('id') - break - - if existing_monitor: - print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") - print("Skipping - monitor already configured") - else: - print(f"Creating monitor '{monitor_name}'...") - api.add_monitor( - type=MonitorType.HTTP, - name=monitor_name, - url=monitor_url, - parent=group['id'], - interval=60, - maxretries=3, - retryInterval=60, - notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} - ) - - api.disconnect() - print("SUCCESS") - - except Exception as e: - print(f"ERROR: {str(e)}", file=sys.stderr) - sys.exit(1) - mode: '0755' - - - name: Create temporary config for monitor setup - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/ansible_config.yml - content: | - uptime_kuma_url: "{{ uptime_kuma_api_url }}" - username: "{{ uptime_kuma_username }}" - password: "{{ uptime_kuma_password }}" - monitor_url: "https://{{ lnbits_domain }}/api/v1/health" - monitor_name: "LNBits" - mode: '0644' - - - name: Run Uptime Kuma monitor setup - when: uptime_kuma_enabled | default(false) - command: python3 /tmp/setup_lnbits_monitor.py - delegate_to: localhost - become: no - register: monitor_setup - changed_when: "'SUCCESS' in monitor_setup.stdout" - ignore_errors: yes - - - name: Clean up temporary files - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - file: - path: "{{ item }}" - state: absent - loop: - - /tmp/setup_lnbits_monitor.py - - /tmp/ansible_config.yml - diff --git a/ansible/services/memos/deploy_memos_playbook.yml b/ansible/services/memos/deploy_memos_playbook.yml index 756ab4f..8b21b85 100644 --- a/ansible/services/memos/deploy_memos_playbook.yml +++ b/ansible/services/memos/deploy_memos_playbook.yml @@ -62,14 +62,6 @@ owner: root group: root - - name: Clean up temporary files - file: - path: "{{ item }}" - state: absent - loop: - - /tmp/memos.tar.gz - - /tmp/memos - - name: Create memos environment file copy: dest: "{{ memos_config_dir }}/memos.env" @@ -136,18 +128,7 @@ msg: "Memos is running on port {{ memos_port }}. Access via Tailscale at http://{{ memos_tailscale_hostname }}:{{ memos_port }}" handlers: - # ═════════════════════════════════════════════════════════════════════════ - # DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. - # - # Every task below is inert: uptime_kuma_enabled is false in - # group_vars/all/main.yml, so they all skip and the deployment above still - # runs normally. Kept because the health-check logic is the durable part — - # when a replacement exists, rewire the push transport and flip the flag. - # - # What was being monitored: archive/uptime_kuma/MONITORS.md - # ═════════════════════════════════════════════════════════════════════════ - name: Restart memos - when: uptime_kuma_enabled | default(false) systemd: name: memos state: restarted @@ -161,7 +142,6 @@ vars: memos_subdomain: "{{ subdomains.memos }}" memos_domain: "{{ memos_subdomain }}.{{ root_domain }}" - uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Publish Memos through Caddy (via Tailscale) @@ -172,117 +152,3 @@ caddy_site_domain: "{{ memos_domain }}" caddy_site_upstream: "{{ memos_tailscale_hostname }}:{{ memos_port }}" caddy_site_resolvers: "100.100.100.100" - - - name: Create Uptime Kuma monitor setup script for Memos - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/setup_memos_monitor.py - content: | - #!/usr/bin/env python3 - import sys - import traceback - import yaml - from uptime_kuma_api import UptimeKumaApi, MonitorType - - try: - # Load configs - with open('/tmp/ansible_memos_config.yml', 'r') as f: - config = yaml.safe_load(f) - - url = config['uptime_kuma_url'] - username = config['username'] - password = config['password'] - monitor_url = config['monitor_url'] - monitor_name = config['monitor_name'] - - # Connect to Uptime Kuma - api = UptimeKumaApi(url, timeout=30) - api.login(username, password) - - # Get all monitors - monitors = api.get_monitors() - - # Find or create "services" group - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - if not group: - group_result = api.add_monitor(type='group', name='services') - # Refresh to get the group with id - monitors = api.get_monitors() - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - - # Check if monitor already exists - existing_monitor = None - for monitor in monitors: - if monitor.get('name') == monitor_name: - existing_monitor = monitor - break - - # Get ntfy notification ID - notifications = api.get_notifications() - ntfy_notification_id = None - for notif in notifications: - if notif.get('type') == 'ntfy': - ntfy_notification_id = notif.get('id') - break - - if existing_monitor: - print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") - print("Skipping - monitor already configured") - else: - print(f"Creating monitor '{monitor_name}'...") - api.add_monitor( - type=MonitorType.HTTP, - name=monitor_name, - url=monitor_url, - parent=group['id'], - interval=60, - maxretries=3, - retryInterval=60, - notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} - ) - - api.disconnect() - print("SUCCESS") - - except Exception as e: - error_msg = str(e) if str(e) else repr(e) - print(f"ERROR: {error_msg}", file=sys.stderr) - traceback.print_exc(file=sys.stderr) - sys.exit(1) - mode: '0755' - - - name: Create temporary config for monitor setup - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/ansible_memos_config.yml - content: | - uptime_kuma_url: "{{ uptime_kuma_api_url }}" - username: "{{ uptime_kuma_username }}" - password: "{{ uptime_kuma_password }}" - monitor_url: "https://{{ memos_domain }}/healthz" - monitor_name: "Memos" - mode: '0644' - - - name: Run Uptime Kuma monitor setup - when: uptime_kuma_enabled | default(false) - command: python3 /tmp/setup_memos_monitor.py - delegate_to: localhost - become: no - register: monitor_setup - changed_when: "'SUCCESS' in monitor_setup.stdout" - ignore_errors: yes - - - name: Clean up temporary files - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - file: - path: "{{ item }}" - state: absent - loop: - - /tmp/setup_memos_monitor.py - - /tmp/ansible_memos_config.yml diff --git a/ansible/services/mempool/deploy_mempool_playbook.yml b/ansible/services/mempool/deploy_mempool_playbook.yml index 76a3eb0..0042fb0 100644 --- a/ansible/services/mempool/deploy_mempool_playbook.yml +++ b/ansible/services/mempool/deploy_mempool_playbook.yml @@ -8,9 +8,12 @@ # about Uptime Kuma — these are just "URLs that accept a ping", and whatever # replaces it sets the same values. mempool_healthchecks: - - {name: mariadb, label: MariaDB, push_url: "{{ healthcheck_push_urls.mempool.mariadb | default('') }}"} - - {name: backend, label: Backend, push_url: "{{ healthcheck_push_urls.mempool.backend | default('') }}"} - - {name: frontend, label: Frontend, push_url: "{{ healthcheck_push_urls.mempool.frontend | default('') }}"} + - {name: mariadb, label: MariaDB, push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_mempool-mariadb/external"} + - {name: backend, label: Backend, push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_mempool-backend/external"} + - {name: frontend, label: Frontend, push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_mempool-frontend/external"} + # One token for all three components: they run on the same host, so the + # blast radius is already that host. + healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" roles: - mempool diff --git a/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml b/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml index c759592..9850e56 100644 --- a/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml +++ b/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml @@ -10,7 +10,6 @@ ntfy_emergency_app_ntfy_url: "https://{{ ntfy_service_domain }}" ntfy_emergency_app_ntfy_user: "{{ ntfy_username | default('') }}" ntfy_emergency_app_ntfy_password: "{{ ntfy_password | default('') }}" - uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Create ntfy-emergency-app directory @@ -52,127 +51,3 @@ caddy_site_name: ntfy-emergency-app caddy_site_domain: "{{ ntfy_emergency_app_domain }}" caddy_site_upstream: "localhost:{{ ntfy_emergency_app_port }}" - - # ═════════════════════════════════════════════════════════════════════════ - # DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. - # - # Every task below is inert: uptime_kuma_enabled is false in - # group_vars/all/main.yml, so they all skip and the deployment above still - # runs normally. Kept because the health-check logic is the durable part — - # when a replacement exists, rewire the push transport and flip the flag. - # - # What was being monitored: archive/uptime_kuma/MONITORS.md - # ═════════════════════════════════════════════════════════════════════════ - - name: Create Uptime Kuma monitor setup script for ntfy-emergency-app - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/setup_ntfy_emergency_app_monitor.py - content: | - #!/usr/bin/env python3 - import sys - import traceback - import yaml - from uptime_kuma_api import UptimeKumaApi, MonitorType - - try: - # Load configs - with open('/tmp/ansible_config.yml', 'r') as f: - config = yaml.safe_load(f) - - url = config['uptime_kuma_url'] - username = config['username'] - password = config['password'] - monitor_url = config['monitor_url'] - monitor_name = config['monitor_name'] - - # Connect to Uptime Kuma - api = UptimeKumaApi(url, timeout=30) - api.login(username, password) - - # Get all monitors - monitors = api.get_monitors() - - # Find or create "services" group - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - if not group: - group_result = api.add_monitor(type='group', name='services') - # Refresh to get the group with id - monitors = api.get_monitors() - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - - # Check if monitor already exists - existing_monitor = None - for monitor in monitors: - if monitor.get('name') == monitor_name: - existing_monitor = monitor - break - - # Get ntfy notification ID - notifications = api.get_notifications() - ntfy_notification_id = None - for notif in notifications: - if notif.get('type') == 'ntfy': - ntfy_notification_id = notif.get('id') - break - - if existing_monitor: - print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") - print("Skipping - monitor already configured") - else: - print(f"Creating monitor '{monitor_name}'...") - api.add_monitor( - type=MonitorType.HTTP, - name=monitor_name, - url=monitor_url, - parent=group['id'], - interval=60, - maxretries=3, - retryInterval=60, - notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} - ) - - api.disconnect() - print("SUCCESS") - - except Exception as e: - error_msg = str(e) if str(e) else repr(e) - print(f"ERROR: {error_msg}", file=sys.stderr) - traceback.print_exc(file=sys.stderr) - sys.exit(1) - mode: '0755' - - - name: Create temporary config for monitor setup - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/ansible_config.yml - content: | - uptime_kuma_url: "{{ uptime_kuma_api_url }}" - username: "{{ uptime_kuma_username }}" - password: "{{ uptime_kuma_password }}" - monitor_url: "https://{{ ntfy_emergency_app_domain }}" - monitor_name: "ntfy-emergency-app" - mode: '0644' - - - name: Run Uptime Kuma monitor setup - when: uptime_kuma_enabled | default(false) - command: python3 /tmp/setup_ntfy_emergency_app_monitor.py - delegate_to: localhost - become: no - register: monitor_setup - changed_when: "'SUCCESS' in monitor_setup.stdout" - ignore_errors: yes - - - name: Clean up temporary files - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - file: - path: "{{ item }}" - state: absent - loop: - - /tmp/setup_ntfy_emergency_app_monitor.py - - /tmp/ansible_config.yml diff --git a/ansible/services/personal-blog/deploy_personal_blog_playbook.yml b/ansible/services/personal-blog/deploy_personal_blog_playbook.yml index 5e3780d..21d8b43 100644 --- a/ansible/services/personal-blog/deploy_personal_blog_playbook.yml +++ b/ansible/services/personal-blog/deploy_personal_blog_playbook.yml @@ -6,7 +6,6 @@ vars: personal_blog_subdomain: "{{ subdomains.personal_blog }}" personal_blog_domain: "{{ personal_blog_subdomain }}.{{ root_domain }}" - uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Ensure user is in www-data group @@ -54,123 +53,3 @@ caddy_site_name: personal-blog caddy_site_domain: "{{ personal_blog_domain }}" caddy_site_root: "{{ personal_blog_web_root }}" - - # ═════════════════════════════════════════════════════════════════════════ - # DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. - # - # Every task below is inert: uptime_kuma_enabled is false in - # group_vars/all/main.yml, so they all skip and the deployment above still - # runs normally. Kept because the health-check logic is the durable part — - # when a replacement exists, rewire the push transport and flip the flag. - # - # What was being monitored: archive/uptime_kuma/MONITORS.md - # ═════════════════════════════════════════════════════════════════════════ - - name: Create Uptime Kuma monitor setup script for Personal Blog - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/setup_personal_blog_monitor.py - content: | - #!/usr/bin/env python3 - import sys - import yaml - from uptime_kuma_api import UptimeKumaApi, MonitorType - - try: - with open('/tmp/ansible_config.yml', 'r') as f: - config = yaml.safe_load(f) - - url = config['uptime_kuma_url'] - username = config['username'] - password = config['password'] - monitor_url = config['monitor_url'] - monitor_name = config['monitor_name'] - - api = UptimeKumaApi(url, timeout=30) - api.login(username, password) - - # Get all monitors - monitors = api.get_monitors() - - # Find or create "services" group - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - if not group: - group_result = api.add_monitor(type='group', name='services') - # Refresh to get the group with id - monitors = api.get_monitors() - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - - # Check if monitor already exists - existing_monitor = None - for monitor in monitors: - if monitor.get('name') == monitor_name: - existing_monitor = monitor - break - - # Get ntfy notification ID - notifications = api.get_notifications() - ntfy_notification_id = None - for notif in notifications: - if notif.get('type') == 'ntfy': - ntfy_notification_id = notif.get('id') - break - - if existing_monitor: - print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") - print("Skipping - monitor already configured") - else: - print(f"Creating monitor '{monitor_name}'...") - api.add_monitor( - type=MonitorType.HTTP, - name=monitor_name, - url=monitor_url, - parent=group['id'], - interval=60, - maxretries=3, - retryInterval=60, - notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} - ) - - api.disconnect() - print("SUCCESS") - - except Exception as e: - print(f"ERROR: {str(e)}", file=sys.stderr) - sys.exit(1) - mode: '0755' - - - name: Create temporary config for monitor setup - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/ansible_config.yml - content: | - uptime_kuma_url: "{{ uptime_kuma_api_url }}" - username: "{{ uptime_kuma_username }}" - password: "{{ uptime_kuma_password }}" - monitor_url: "https://{{ personal_blog_domain }}" - monitor_name: "Personal Blog" - mode: '0644' - - - name: Run Uptime Kuma monitor setup - when: uptime_kuma_enabled | default(false) - command: python3 /tmp/setup_personal_blog_monitor.py - delegate_to: localhost - become: no - register: monitor_setup - changed_when: "'SUCCESS' in monitor_setup.stdout" - ignore_errors: yes - - - name: Clean up temporary files - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - file: - path: "{{ item }}" - state: absent - loop: - - /tmp/setup_personal_blog_monitor.py - - /tmp/ansible_config.yml - diff --git a/ansible/services/phoenixd/deploy_phoenixd_playbook.yml b/ansible/services/phoenixd/deploy_phoenixd_playbook.yml index 0df1223..b9cf243 100644 --- a/ansible/services/phoenixd/deploy_phoenixd_playbook.yml +++ b/ansible/services/phoenixd/deploy_phoenixd_playbook.yml @@ -9,6 +9,7 @@ # decommissioning — its systemd Environment= was left empty. Leaving it empty # preserves that; the check still runs and its exit code is still the answer. # Set this to plug in whatever monitoring replaces it. - healthcheck_push_url: "{{ healthcheck_push_urls.phoenixd | default('') }}" + healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_phoenixd/external" + healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" roles: - phoenixd diff --git a/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml b/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml index d03363c..282fc2e 100644 --- a/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml +++ b/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml @@ -6,7 +6,6 @@ vars: vaultwarden_subdomain: "{{ subdomains.vaultwarden }}" vaultwarden_domain: "{{ vaultwarden_subdomain }}.{{ root_domain }}" - uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Create vaultwarden directory @@ -84,128 +83,3 @@ caddy_site_name: vaultwarden caddy_site_domain: "{{ vaultwarden_domain }}" caddy_site_upstream: "localhost:{{ vaultwarden_port }}" - - # ═════════════════════════════════════════════════════════════════════════ - # DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. - # - # Every task below is inert: uptime_kuma_enabled is false in - # group_vars/all/main.yml, so they all skip and the deployment above still - # runs normally. Kept because the health-check logic is the durable part — - # when a replacement exists, rewire the push transport and flip the flag. - # - # What was being monitored: archive/uptime_kuma/MONITORS.md - # ═════════════════════════════════════════════════════════════════════════ - - name: Create Uptime Kuma monitor setup script for Vaultwarden - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/setup_vaultwarden_monitor.py - content: | - #!/usr/bin/env python3 - import sys - import traceback - import yaml - from uptime_kuma_api import UptimeKumaApi, MonitorType - - try: - # Load configs - with open('/tmp/ansible_config.yml', 'r') as f: - config = yaml.safe_load(f) - - url = config['uptime_kuma_url'] - username = config['username'] - password = config['password'] - monitor_url = config['monitor_url'] - monitor_name = config['monitor_name'] - - # Connect to Uptime Kuma - api = UptimeKumaApi(url, timeout=30) - api.login(username, password) - - # Get all monitors - monitors = api.get_monitors() - - # Find or create "services" group - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - if not group: - group_result = api.add_monitor(type='group', name='services') - # Refresh to get the group with id - monitors = api.get_monitors() - group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) - - # Check if monitor already exists - existing_monitor = None - for monitor in monitors: - if monitor.get('name') == monitor_name: - existing_monitor = monitor - break - - # Get ntfy notification ID - notifications = api.get_notifications() - ntfy_notification_id = None - for notif in notifications: - if notif.get('type') == 'ntfy': - ntfy_notification_id = notif.get('id') - break - - if existing_monitor: - print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") - print("Skipping - monitor already configured") - else: - print(f"Creating monitor '{monitor_name}'...") - api.add_monitor( - type=MonitorType.HTTP, - name=monitor_name, - url=monitor_url, - parent=group['id'], - interval=60, - maxretries=3, - retryInterval=60, - notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} - ) - - api.disconnect() - print("SUCCESS") - - except Exception as e: - error_msg = str(e) if str(e) else repr(e) - print(f"ERROR: {error_msg}", file=sys.stderr) - traceback.print_exc(file=sys.stderr) - sys.exit(1) - mode: '0755' - - - name: Create temporary config for monitor setup - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - copy: - dest: /tmp/ansible_config.yml - content: | - uptime_kuma_url: "{{ uptime_kuma_api_url }}" - username: "{{ uptime_kuma_username }}" - password: "{{ uptime_kuma_password }}" - monitor_url: "https://{{ vaultwarden_domain }}/alive" - monitor_name: "Vaultwarden" - mode: '0644' - - - name: Run Uptime Kuma monitor setup - when: uptime_kuma_enabled | default(false) - command: python3 /tmp/setup_vaultwarden_monitor.py - delegate_to: localhost - become: no - register: monitor_setup - changed_when: "'SUCCESS' in monitor_setup.stdout" - ignore_errors: yes - - - name: Clean up temporary files - when: uptime_kuma_enabled | default(false) - delegate_to: localhost - become: no - file: - path: "{{ item }}" - state: absent - loop: - - /tmp/setup_vaultwarden_monitor.py - - /tmp/ansible_config.yml - diff --git a/ansible/site.yml b/ansible/site.yml index a5f8576..8e9151e 100644 --- a/ansible/site.yml +++ b/ansible/site.yml @@ -29,6 +29,10 @@ - import_playbook: infra/400_host_monitoring.yml - import_playbook: infra/401_service_monitoring.yml - import_playbook: infra/402_public_monitoring.yml +# Registers where the per-service probes report. The probes themselves are +# deployed by each service's own playbook further down; the endpoints must exist +# before the first push arrives. +- import_playbook: infra/403_service_probe_registration.yml # 910_docker says `hosts: managed`, but only 5 of 11 managed hosts have or need # Docker. Left out until it has a [docker] group — see the note in PLAN_7. diff --git a/ansible/services/ntfy/setup_ntfy_uptime_kuma_notification.yml b/archive/uptime_kuma/setup_ntfy_uptime_kuma_notification.yml similarity index 100% rename from ansible/services/ntfy/setup_ntfy_uptime_kuma_notification.yml rename to archive/uptime_kuma/setup_ntfy_uptime_kuma_notification.yml