counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

watchtower: remove from the estate, and with it ntfy

Browse source 
watchtower is being destroyed. Removed from [vps], with its host_vars, its push
token, and the six Gatus endpoints that referenced it (liveness, disk, two
systemd services, the ntfy DNS record and the ntfy HTTP check).

ntfy went with it - it ran nowhere else - so services/ntfy is deleted,
subdomains.ntfy and ntfy_topic are gone from group_vars, and the ntfy playbook
is out of site.yml. ntfy_topic already had no readers: the three infra/4xx plays
that used it were deleted when their checks were superseded.

Two things this exposed.

services/ntfy/deploy_ntfy_playbook.yml was pointing at the WRONG MACHINE. It
said `hosts: observability`, which resolves to the host `monitoring`
(64.226.70.190) - but ntfy ran on watchtower, and ntfy.contrapeso.xyz pointed
there. Running it would have installed ntfy on the new VPS. Moot now, but it is
the same stale-identity failure as the rest: the group meant watchtower when the
play was written, and nobody revisited it when the group changed. Watchtower was
in [vps] and NO role group at all, while running caddy, ntfy and Uptime Kuma -
nothing in the repo managed any of it.

More seriously: ntfy-emergency-app on vipy (avisame.contrapeso.xyz) sends its
notifications to https://ntfy.contrapeso.xyz, topic "emergencia". Destroying
watchtower breaks it, and it is an EMERGENCY notifier - it would fail silently
at exactly the moment it matters. That is NOT resolved here, deliberately:
standing ntfy up elsewhere, pointing at ntfy.sh, or retiring the app are all
decisions, not cleanups.

What this change does is make the break impossible to miss. The URL was derived
from subdomains.ntfy, so deleting that would have turned it into an undefined
variable buried in a template. It is now an explicit ntfy_service_url in the
app's own vars, still holding the old value, with the three options written
above it. The ntfy credentials stay in the vault because that app still needs
them - the vault was restored from HEAD and only watchtower's push token
removed, rather than re-handling the plaintext.

Verified: no reference to watchtower or its IP anywhere in the repo; Gatus down
from 91 to 85 endpoints, 85 UP, 0 DOWN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
counterweight 2026-09-14 11:26:47 +02:00
parent bf3d21fef7
commit 85040d5f67
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
10 changed files with 171 additions and 282 deletions
Download patch file Download diff file Expand all files Collapse all files

5
ansible/group_vars/all/main.yml
View file

@ -26,7 +26,6 @@ backup_pull_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfIixKMhA9z+Nvyx6T
subdomains:
# Monitoring
gatus: status
ntfy: ntfy
# VPN infrastructure (spacey)
headscale: headscale
@ -49,9 +48,7 @@ subdomains:
# DATUM Gateway dashboard (knots-box, proxied via vipy)
datum_gateway: datum
# Read by plays targeting managed, monitoring, vpn_control and edge - no one
# group covers them, so these are global rather than group_vars/<group>.
ntfy_topic: alerts
# Read by plays across several groups, so global rather than group_vars/<group>.
headscale_namespace: counter-net
# ─────────────────────────────────────────────────────────────────────────────

309
ansible/group_vars/all/vault.yml
View file

@ -1,157 +1,154 @@
$ANSIBLE_VAULT;1.1;AES256
30373030336530356533636231303630303132666434393339663833366639366234313434333635
3763366235343633303836383563333734643766346336390a383732313137623263343836616531
35656262386162346434343462396166653435633330613366633263356638373061643833386336
6438373137326563380a353962396135393365376133626538303862396236363330636564323538
33616565363066633234633330386338316133613931613931653963356562323336396437313130
33316331333166393161376635306237363033633535613761373762393761363165623462336466
63663265376532663262323463643462336136373639316565396436653966336461656164316135
64303235386236316136623063653133363961303733376166363462303565393761636632666230
66333263626537343935383765303565626237346235356338373063633133326465653133366362
62326563666531343166663563393932393938363663393132616561336363643037363735393233
38336233346138356262616533643835396230663563623237616461626265666339613161616331
34353263363065646336663538663561373536656364623061633863643137633230643931333662
34386433326661336362636466386566636561623339383438626530333431313039623037616430
62333832363531323363646238363733316237646133343337363434373465353463363935323233
37326335343333636337626337643763323336623238636463623933646663356165363062383835
30636630303534636436326530313030626565393439643238353433396638363133353865643135
39666464323537383435636636303930396338333332363264306664383538666236616436353939
61316564623964613839343533316438393337616363633033636231313937366139373031616236
61373763303231376437313332343438376132633361353661336232333966313338373262383663
36363931376132333763363332313534616133326437303637343139383861353138623465623736
62643062396536633730663130353730356637353533616635343166643636616163343832363866
65646266383062373635306239323565383039646334346535653962316434393365646461653366
37396163396362313535393138663833653435353934623432616538663565353165363732316662
35316332363030363963666136653437316639303766376431333738343061663835306339336337
64346137313763656531306636333030323162646533633735646435613566616532363066353361
32326439633432656633303732626361636238323735636631633732383635656436373435343931
64356437326633326163306430306432326134373366616364656361383133656531653333633164
37653934643061343237376664643331623633653437336534313261643765303134626536393938
33393161363539313130376132313864666539363865393035626463313539376635393135633565
35306435636363613735343136663939663435653135376337643034396634623135393039366265
36326332386564303233623863363236356535323538616237303262613261363964643839363837
63336536613139333939343964323835626433373464343730353864303532366535343636663362
34333935316630323137366130356135643962373532346334333931366139356434316431393234
61386632353434323432306130386239663931306538323335333231366234383132623936343465
32393765316136663862663334616234326165336364316137396234343532643639366537343731
33383938363530323337333838643835653238376532376431393439636266306634386233383937
64623230353533643362633563663537343732646261353763366363343231376562336565333362
65623665326136383764373761626461653238303335616463636333383235393037323530636630
65333036653062386332653433366532383132376362356264373432653564316530656163396161
34386133326133393638653961383032323830366232346134323663653436383034356334643137
37663062396534616563363166333535336466373335613733323864363736633263646438336662
39643333366565383130346430623637666132656330653833393836313566393461656638653938
32623664363835376366353532646564643235356637363432363164353837643066643932383163
34643232623734613738613638306664326333346439383830326662653934636536386166323239
63626264613661336663316636353436643262353733303666663866346431353533353435346461
34343763316364303061353030626431613434383062363737356131366234626137656331383433
61653339616336376562326638343462343435613632656632353532376438623462656266326539
35336465366133386534343466323936663934306363353462356530323531383961633336393234
37366266343831386462626539306637623165363066616164393635373631393737613830363661
34656530336634336531373830613039616231343238653637623532386538626365386364353930
32396535643435306231386636363562623239393461393237303030353361633639626632653562
31663261653663356266376264623762353261613163303430623738343939336438333532353436
62343465353033343034373837376538323239653064623135613766343463373436663236616366
30313062663861646331316362653230653139623937343538346230613832623965366666323365
35623635646263396436346162333835343335623364393037353366343537336462636163303761
37346663306339333264643034393630613833343163306430396333656637316662616439343136
35336366306535333733633465343538383462616631326433666663303638633837313065656666
34326136363433303264353331393133626639343166303364343065333266646439663463353833
62346637363265643736616238663033666563326462633562643530383862616265306439376439
63353432343534303433353138663461303165313433353866303838346338363761376333366138
34353035356132383836313134643363636532353834323438313933346534333238646532326263
30663964326564623164306664323265383134616165353634653261383931326663356137363636
36383663666537373863623532376165343334633031626661306664663139616637636162373838
32663863393765343431613536386133653731363266663566303665316435363830303035656230
31366139636332336531326131346237653732313337343336393463373438373531653262363035
39613739323832353332643030343861373836653064326139343965613563363166363236343733
39653666333438613131396566623237643765393435313163333031383461303366396439353437
31376562323231386133356231343366623363396437323866326234613034353664323163646631
36386436653465306665646130616563646564343764346532363961663762303132336137356231
61376565333036313033633732616430343266356435376434386266306432633633316361656538
66393266383735336632653834373561636261663738333039653934636434343165353065353566
34616431616430306234313032346235383065633734323632643065613634343834386664313336
33333861306362333032346633636630326562323430323863373532613831313931303164303438
36353831623262663030313564616634376235383666656434373261643464363264653831386534
63663931643733313239323138393936373435303263663662613266306162373363353334336430
62393835363263666136646631653738393432346331353538306666663864323233386665306332
39353839313731313832333139613830396539306133363236653138643161323337356534643037
38626636353465636166313965346364663238363032383537306532303332643839346230666564
66613261396633346365633730633738306438326234386438373234393537353635626664616239
35363737363537326132376638373839326139643135333234333764626162343766623531643265
32656464383064626566653332323366306138373537663634333833613932666264306164666435
39623335323163636135313361376231316231626335343765323261653134636439643561663463
33376130356362316334386162306333333038316664636464653463313835356461386464653035
38626230376339663361306161386332656230353737376133306464626466653038653266646139
62313937383262633938633735393765616365646165396436653434303835666162373164333938
61643432343237386133316433613030633638633731323936393562376139353033643562633835
36393837383261663761393763613039623263643266303637613461626462313162613762383535
33373064646533366136376563383663633331373161646534653330653566303332616262386564
30313330346463643032363464346537613430633163306365313866373031393965383134323031
63653238343338613033633336366235623332646239613235643637313666616263313833366362
66666633393232303639363966306230663731353730333264663732386235653633303336316139
33646263303164613437343735616362636364376261323364616131383433633864323132353063
32646530323838633962613834663564326137663466623935323665343132653037386339613961
61666363323763636361396330663537386630306637663638343565653437623966313738306366
37633436373434396231396461656634316638393336653964396266363532343864346264323830
32616461353630353730393639306231363662663034626434316661653936636262393761633934
33393863383461316532373839663432336666356332393562393634386533346262306331643733
62373162643763616637316233636163636363363639303662313630336233353162343437356533
30303339666136343665313038643930383539313035373566336139643766353534386232343330
32633836356435326332666266393237353162646537376564656236353138386531323032626636
37373963666666346462343938656466343766323434373064363736386134656232636631653535
38643739333230313265643132356430393035656662356465623836323038663237383439396137
33383335356433383030393131666233663832643638646537363665396435323261613732303363
63653634666164636632306462346166623237303936326435326431636630323562613233386462
64376562316362653937646266366232653733663764633738303162616538656661326261643139
30376233633835333637356134626266376566393962353639663039323439616639316465353038
32373136643162633836663937373664306665643638366237653631313631393934643135393261
35353338393761383431326631633166653932313335346165653364316431333831306135643339
63316538343963336237653366343836343330656364373661343866656235303931633432313966
35393631333263363533316537616132323038303230653838326266336434336532346439626537
30656334316239356261386633393635643563336230326362303034633235323830343435646136
63333237346164323763633665636234313263393366636233393538353335383933636331656261
39313733383238613634363139326536616237353031666232663161653763326162386464626533
39633931353035356561643564656637633163396561356430663636313231333962363765373536
62613366616538306331623537613564353437366239616332626237623466613931653339333431
39646239383438623063663063333861356536326637656337303561346432333539656431373733
31316461343636313430303434336535353430353932373536366162333938646464653763356237
39363938373065663732393862383031336438396164366664323130343734323130306662343138
35383734663465303034393561306264316139623265386134323162373734393364393230333234
31383336363832373962613535346136353037383065363066653935373435383037313764636266
65383133333433326662626639356162303261393866373732366665353465356534333431633639
39303539346266316132643537393564346562353533363438376363613036613739633939353564
37313764316661663939396664376538626437366534383630613030323465396535366537346263
31636338353233306333653962373763353439346337323839633238636336653439386362333533
31653264343435633432646633386531386139366461646133396334326631333632393462666232
35383735356232336335336435343932613433393530653633616466373965383935383861366136
31303936616333396638326465633531373164626232356362613130376434303633366238336666
37643035303932393265363037646430343866323163386234633739393938656530646533633163
61653135633436653566303832383564663462353235353134313033616364636561643663383835
30326561653934636164636363363736346630316263616664386233373535356339336239653333
32336362363033616337663936623331663134656665643739626637353739666231643766346134
63353631663631643436633935306261373939323530363639666366373531626231366138643766
34623137643233333232303339323464333566633038333539356337306134383330353632383965
34626461316261363830376664343336666132376365333635666131653933323562666635353763
66376335663861623332623436333161353161336337313136613531646632363531383138343130
39306330336239393965383939333765633532376539653661353965373030313038666161313265
37653164343832333937356434646230623736646561623561643135626531343063626636663333
30363937613834663434313038653739613933656534626532326136323336386164366339313830
35396130386634666363656237373835313563343961633838383766663933376439326364616530
64383136623739623137363562666431333565643130346166663531373738643761363337333536
38313866363339393066643339653734316266393037396432356138303137343536313134623434
62623362383863303539663264386430353436613261643865666534356538626538623630643766
66646266336238396230613838626161336637313564303465663034653232306363306430633438
35303364356531636565303965323539393233663566646562383863323461396166623032396338
36346537343633643864323163623464623539616434376137313538646333666339626437656330
31326463666335626361616566383065363264356663323435646337663566636436356336376531
38636466386539663937326462636464613638653833666263333134396236313432643030366138
66326364623435356633666463333066373238363864356230633634353764616331303163636266
36306239323561326134393665643031663362636535656139376237383130616663316362326366
61623236356562346233643332663261626333646638373262613664623935626266306230356164
32336363383538333935333062386366653839303139366565343933346266623637363534393335
31346431313631373535306138633930383536613636393034343434623664376237373066633036
30323864613430333239326366613439613632633933656631356237333930303330326264646563
38303066323563303262663530326637663964373039336431353339626436363335326466313262
35386533633833363038636339663265626261373464633565666166633135333461363165353931
63633332613966663133
38396132363836373161663662323635613764643231316132333265396363353833383839613233
3564323734323561366536303932636435373339363233650a343964613465353137396663666363
63646331366162336537333035386263623264383566646334613332376561613231643235313565
3638326434613462640a653364646361626636376464633734343332393030393232616362633532
61313161643939383930303362363966326161313237346565323532626135643361633936313135
38396534616239383735353532613834373564336537323531303533323434336166326263376238
35303032336237666235396562313738363130326633336338306632353161336366346332393465
61396163366432323335393362613539313661333431656465336662323336393330373332383834
39636230313566663630656531323530653130643362363562326465643665326335323036346237
32303439356432316564313730653332323731343235336535373462326162656638383735396135
36633264613136323866643162353161306334666463303565333936653861643635336162623831
65386231313465363864623035623364303039323961363237393637303133386530626266663435
37336662643632666164386264646330363135623565306433646437333535363839663330663533
64373231386231643863303933326132646364333339326431646531643962356663326363643734
66346239663438363666613636336430336465623866643861666564353739623238376135346431
38346234333364646230616632646561643638656535306630326332626136396538333534653061
33643562326436636664343339643834323466313131323462653234303834373635313831353538
35333739346563646163333937643232343866383337653462653530666165643361303332363364
36326333333234343761386234666534323430323734616266326231386136336463636231346237
33356361333562656666393630323530333731613964643966383065663635613734383066333532
31366533373533656666666336363666313839316133333062356432633038366363623935373566
33356563663933653539386263663530363239613461383831646262353165343638326663383364
37386330356539343131353633633233323738396339336230666430363965353536343031646163
38306162643733326563373835396264343433666335366136353439356333646131373764336438
61393836303038306132383363633838333665343665303136316131616335643337313362306661
37373137653830636635366266323538393338333463333963343931343433323437663661313366
36616164346265626637616665663461336163343666623162663637633037666564363533623833
35386537306366636533633930326339343136303839306539393866306462303632363163313535
64326530323866313539623661636266663933343165653662373331663334376333393834303938
30373233323664663935343364363731636334343065643662663266343366353031336439383766
39313862616230346661313632613662613335666234333432336262663033633437383565383336
35316365643030616331643333356361373236643731643733386237666430366566303933326563
34336662623431333539616337303739373035313165396535393435383235313633616632616261
37383633306235633939326339303861346335313361316438373630383863643864376462373464
64663631656165316664633336333634643939333464303266366438633039653038356439393565
35343237303533363032396365346232616131306666353766386433393737626639656139373332
65346336373936393866666236346262623666666435363065386563613431626537636364396238
64373133653132366538626262313434353930366338383866653139376538666136383363333666
39636538323437616633356337626130356164363663333637333232663435303537336233643135
64613638643564653431386634376538643564333438343835376633626230303264316337376434
66316231386533313364343131316162623334373065626533303239383439326134343830376438
34333831333035383830386436643239383138633166303033363563313562613564353538373536
33343962346439616439366463383236373437613132326162323437343231666231396332623038
37643932323538633365393033386462663735333034326332663461636363613932346637623638
37323534613439373865646139656462303266373636356630306431323065656533663838303765
30306534343965366566663337313263643832633861336334313835643738656334653135626563
64353062343561626665613162346236393731623533663431333866613932653331613361333039
65336337666264386334373039343965323065626334353030336336396136633363323339313031
62633932613934623065306638656135393438383361376465653431316261633633353031323066
32323466323861316338343137306333326137376434363735393664353761303265653165316634
66623866373730646138333162663830363063326231653636643633623936313164363065396661
64336263333234356263636630666264393734343732613231333331616230333662356164393539
37616533356566323936386331623133666335613765303732313439376461363837623136386638
38313666633436316336396337643666623735396562326538616564363066356136306632313138
36643333346433366233636463626538313465666334373561666433396137323532366662393363
62313434346636386566656437313130663836306265326463383264646564643538333738313436
65343531356138353632346434616134316261306162636439666666326439373836333761333332
30323836633438383166633364366365363561333731306435363433623333316665646334623061
34396131666438613938343364663237393538313632356166346432346530386131623435643566
30613233313739323562373332343537323665623330353135373735663035376138383931333938
30363066303932643465633534663863336339356431393335343261333235396130616165636538
32643335373137373638636537373635616133646333653230376633656561323533376136376564
61306563653736656364383935326635363661636663626530313938646130356132373037373135
64373034656232623530386435633838363262376464376361323161343163373561393839336165
34323137363837616466326364376336646636396364393464626565623934636231643161343032
61616663626130313764393735633866393566633533343737633035623138613662313335656136
61303132333935613236336164396330383136303164633033356638643835313065393836326236
36356235623838656534363639343261623939633864663631343363663262396231343562396530
30653165663332626434636231303637356538666337643361653338303330633932643763323034
66366538303939323232383631383562613732313065323933343761656535366134323164396432
65663339366631623738323135633965623966613433376132363930383061613434386537373332
39623136646638393835646163656364316665646464356535343839393966363631383734633536
62393832346632323330356365393133346134653837663165353834313931356236373664656234
39363431646239323665386639646165303836303731383063663266373236343436633163636535
31393537653962303236383930613339613232643864356230616664313038396266333635396436
36373930646435333863626333333439626663663930663730323236666233386231303730653132
36613763303739376135653531326637313437623338343232303464303639633662373138326134
33616632373031303139313439383465336565323837656632336532373737396230613837376564
63663334363636666534616265623062366132376234326132393732336563313165386334626264
34373934323930663832623338393964353964316132376262353662643534303436643537646361
31663931613164646435656462613433356366633862636461633938343039353531633364373232
31623131616534653533656566613038346133343266373637346663373739656161343765666636
61656466636563643464616565643837623238636466316538636565626261313165346238306261
62346264313739666338376439666633363033336435323932613664373234623931323962393035
33313639626438656134363565396361613965376662386139366163383238346630623064343634
38633962323232666636303830663538333737376437393465626338643632396238353863646263
61363764346330663866396337626237653630633335363436613863613231356262383039613831
62356465626564386537373134663536396231326136376234393966303065333264376630316633
66303262383237386632656231663835383534326333633862383033356263306436623966383161
65663530303836653235616466333736636463323065626335356333373334376337653637666133
30363232383533646331323030303434633162643733623339346132666564396663343131303233
62373431633635356235396130393330643463663239616365663930366138393930343334343432
31616431666361353636316163313430633730366462616332333830353937616466343265343165
36366265626363373164373463353561323437303936393337336238393137633931323537653666
66346231363636326636333736383637376535393961303731386535386138633462323030346330
64353936383162356433306164616264366164623731373733626261653238306362623264666633
64316366393765356632323966343634653235636162353636306337383864326264343434613961
65656433323164363738313535636637383365636435343933333265653333326536363864303866
38396263626166316537326534363637393132363334616134653036663830343764303737383663
31616639373330363365336231333938303138623165666531336336353133653263313263343231
39666639383931653231373535396365386637643738346438333961393138633961376134303664
39336331646236396634356337363336316537643133653561363937373963636238323837303833
33396164656434666239393163363638373364383839633364393137313765613531626337316532
33386465613034663630323638653839666264373235663364643461333238376138303765643364
63623764363830356264306363613266393930376564616438636438373063323030633032326333
66316235643036376165323035386164633536396365326333346638346133336235616132663532
38396636376539656332613963646264316637616235316131316232623131386662333164333534
31366534303166383062316263393833326165346333623830653266646637663364373365626335
61623837323934343131353336393163303831613832303530356639393465633461326566363138
39363730353862383365386339353536366537363933663566306132626135336339646533323763
65616664393166666461643038336232303934306338323331366233643061373532623163663931
37343839346637376337613438616131633239623234613539323138353233343238653661626462
34353666643230633562323631303732373332616335326131396131653038643531646562646431
33626130343834396439323138393162363831393534383839663638663838633061666534336138
35313234653565316437336439376563666138653661613062643262656233316262343736363235
32623466626366343965373638323239393933386434633531303363313061313832663166653634
39333835303066303231363663666461323434363237646133323661386663633939313965363864
38303861383565303766306533633132376632373234646338306264343065633631303935313266
30333836616164363735363934386165663839393731633039383538623833353632373636656436
62343934363834613464626364303434393262373365643130373437393730393964316130613565
35373938306436666364646136306166663132393736383862623635316265386361663333356431
35336363623535336631366531316661333130393034383566613138653834383333623437356139
36353434626531616361333138386338636132646533326565336534643632316363313732336233
38656134383565303132643161393032333334316265343862666532303438653161626665326632
31343435393136313865356166363165613730373633623963303633343165613230393430633831
31376361656361666434343866353037333834353735396566643935393439343334376461323133
61663835656137356665623265376664336332313432636664313735643739626333383566336331
35306539386433633636356636383734636633666137653334333162386564643464306538366461
33323130656331313834323839303736366436633634356261363335343231303331353339313031
61613563383937623162616133313633653833666634643566643766336533303566383561633237
61316562353933333666323037376334326430623035643630323565386532313530383835343265
61616165396664616332313262666534396537356564656236313864326533306338373362613034
38623162336138366262626538366333373830623634616261353764666563663138663862656339
65346137646263393064393536613963666434373663636336373531316464336264313061316439
39623832666536356334383261653431343837663163623035366162396465646239613130643837
64393735333438376262323834313330363434366163653833376334666666653535633330636535
66346665643332393235313062313164333030353265386632326431333036333861356231666531
63323966623361343363623561386633363734323864623961393832653037353766326637663735
63396565326632303366396539396435626235643833323864663731396636313237373832646566
66333839643734303734376338353037393266353232316264353736346338646262386636616335
38313838363064643739633065343535643762626165343862646362613462386532356166383437
38646130313462393863376339336436336630643061363130663831313532663466363936343934
31656230323564303638653832626364633631336336613663356562636436643336323238393436
63343462366436653365383766366262373332313339626431326538333562346438303730343937
33326365636631633365336332636638353663303633623137643764303930656336323963643835
34376136353532613938326432663835363530653264383561353533663863376661636137336437
61353365333661623038363535643035393335613632656330333938623335343436663763646362
63633130373530386235663661613335323531363234343465313861656639646461613936336134
30613937376666623761626438396635313563306564633433613330633664356636636565613166
32633430646139383362623133346435633063633466656635333034343937643639346536393134
39383664383234643265323563646465373636303261646331366439666466343636333330633131
37363963373361613331663263623737373464663831326630663432316336336163343866663834
6630

13
ansible/host_vars/watchtower/main.yml
View file

@ -1,13 +0,0 @@
---
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- caddy
- ntfy

2
ansible/infra/402_public_monitoring.yml
View file

@ -22,7 +22,6 @@
# ansible_host - if a box is renumbered, inventory is the one edit.
dns_records:
- {sub: "{{ subdomains.gatus }}", host: monitoring}
- {sub: "{{ subdomains.ntfy }}", host: watchtower}
- {sub: "{{ subdomains.headscale }}", host: spacey}
- {sub: "{{ subdomains.vaultwarden }}", host: vipy}
- {sub: "{{ subdomains.forgejo }}", host: vipy}
@ -42,7 +41,6 @@
# there would go green precisely when the auth broke.
public_sites:
- {name: gatus, sub: "{{ subdomains.gatus }}", path: "/", status: 401}
- {name: ntfy, sub: "{{ subdomains.ntfy }}", path: "/", status: 200}
- {name: headscale, sub: "{{ subdomains.headscale }}", path: "/health", status: 200}
- {name: vaultwarden, sub: "{{ subdomains.vaultwarden }}", path: "/", status: 200}
- {name: forgejo, sub: "{{ subdomains.forgejo }}", path: "/", status: 200}

1
ansible/inventory.ini
View file

@ -1,6 +1,5 @@
[vps]
vipy ansible_host=167.172.107.33 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
watchtower ansible_host=164.92.239.72 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
spacey ansible_host=64.227.112.128 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
monitoring ansible_host=64.226.70.190 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua

13
ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml
View file

@ -6,8 +6,17 @@
vars:
ntfy_emergency_app_subdomain: "{{ subdomains.ntfy_emergency_app }}"
ntfy_emergency_app_domain: "{{ ntfy_emergency_app_subdomain }}.{{ root_domain }}"
ntfy_service_domain: "{{ subdomains.ntfy }}.{{ root_domain }}"
ntfy_emergency_app_ntfy_url: "https://{{ ntfy_service_domain }}"
# ⚠ UNRESOLVED: this app sends its notifications to an ntfy server, and the
# server it points at ran on watchtower, which is being destroyed. This was
# derived from subdomains.ntfy, which is now gone with it.
#
# Until an ntfy server exists again this URL is dead, and the app fails
# silently at exactly the moment it matters - it is an EMERGENCY notifier.
# Three ways out, none of them automatic:
# * stand ntfy up somewhere else (the monitoring VPS, or vipy)
# * point this at the public ntfy.sh
# * retire the app
ntfy_emergency_app_ntfy_url: "{{ ntfy_service_url }}"
ntfy_emergency_app_ntfy_user: "{{ ntfy_username | default('') }}"
ntfy_emergency_app_ntfy_password: "{{ ntfy_password | default('') }}"

6
ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml
View file

@ -14,3 +14,9 @@ remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_h
remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
# Where the emergency notifications are sent. This pointed at ntfy.contrapeso.xyz
# on watchtower; that host is being destroyed, so this MUST be repointed before
# the app can work again. Left at the old value so the break is visible rather
# than silently defaulted to something plausible.
ntfy_service_url: "https://ntfy.contrapeso.xyz"

97
ansible/services/ntfy/deploy_ntfy_playbook.yml
View file

@ -1,97 +0,0 @@
- name: Deploy ntfy and configure Caddy reverse proxy
hosts: observability
become: yes
vars_files:
- ./ntfy_vars.yml
vars:
ntfy_subdomain: "{{ subdomains.ntfy }}"
ntfy_domain: "{{ ntfy_subdomain }}.{{ root_domain }}"
tasks:
- name: Ensure /etc/apt/keyrings exists
file:
path: /etc/apt/keyrings
state: directory
mode: '0755'
- name: Download and dearmor ntfy GPG key
shell: curl -fsSL https://archive.heckel.io/apt/pubkey.txt | gpg --dearmor -o /etc/apt/keyrings/archive.heckel.io.gpg
args:
creates: /etc/apt/keyrings/archive.heckel.io.gpg
- name: Add ntfy APT repository
copy:
dest: /etc/apt/sources.list.d/archive.heckel.io.list
content: |
deb [arch=amd64 signed-by=/etc/apt/keyrings/archive.heckel.io.gpg] https://archive.heckel.io/apt debian main
mode: '0644'
- name: Update APT cache
apt:
update_cache: yes
- name: Install ntfy
apt:
name: ntfy
state: present
- name: Ensure ntfy cache directories exist
file:
path: "{{ item }}"
state: directory
owner: ntfy
group: ntfy
mode: '0755'
loop:
- /var/cache/ntfy
- /var/cache/ntfy/attachments
- name: Deploy ntfy configuration file
copy:
dest: /etc/ntfy/server.yml
content: |
base-url: "http://{{ ntfy_domain }}"
listen-http: ":{{ ntfy_port }}"
cache-file: "/var/cache/ntfy/cache.db"
attachment-cache-dir: "/var/cache/ntfy/attachments"
behind-proxy: true
auth-file: "/var/lib/ntfy/user.db"
auth-default-access: "deny-all"
owner: root
group: root
mode: '0644'
notify: Restart ntfy
- name: Enable and start ntfy service
systemd:
name: ntfy
enabled: yes
state: started
- name: Create ntfy admin user
shell: |
(echo "{{ ntfy_password }}"; echo "{{ ntfy_password }}") | ntfy user add --role=admin "{{ ntfy_username }}"
- name: Publish ntfy through Caddy
ansible.builtin.include_role:
name: caddy_site
vars:
caddy_site_name: ntfy
caddy_site_domain: "{{ ntfy_domain }}, http://{{ ntfy_domain }}"
# Raw body: ntfy needs a plain-HTTP listener for its CLI/app clients,
# with only GETs to the docs and topic paths redirected to HTTPS.
caddy_site_body: |
reverse_proxy 127.0.0.1:{{ ntfy_port }}
@httpget {
protocol http
method GET
path_regexp ^/([-_a-z0-9]{0,64}$|docs/|static/)
}
redir @httpget https://{host}{uri}
handlers:
- name: Restart ntfy
systemd:
name: ntfy
state: restarted

3
ansible/services/ntfy/ntfy_vars.yml
View file

@ -1,3 +0,0 @@
ntfy_port: 6674
# ntfy_topic lives in group_vars/all/main.yml

4
ansible/site.yml
View file

@ -56,7 +56,6 @@
- import_playbook: services/vaultwarden/deploy_vaultwarden_playbook.yml
- import_playbook: services/forgejo/deploy_forgejo_playbook.yml
- import_playbook: services/lnbits/deploy_lnbits_playbook.yml
- import_playbook: services/ntfy/deploy_ntfy_playbook.yml
- import_playbook: services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml
- import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml
@ -80,9 +79,6 @@
# infra/nodito/30_proxmox_bootstrap one-shot: bare-metal bootstrap, run once
# infra/nodito/33_..._cloud_template one-shot: builds the VM template
#
# services/ntfy/setup_ntfy_uptime_ creates a notification channel INSIDE
# kuma_notification.yml Uptime Kuma. Kuma-specific tooling, not
# a deployment, and Kuma is being retired.
#
# services/vaultwarden/disable_ deliberate manual actions, not convergence
# vaultwarden_sign_ups_playbook.yml