watchtower: remove from the estate, and with it ntfy

watchtower is being destroyed. Removed from [vps], with its host_vars, its push token, and the six Gatus endpoints that referenced it (liveness, disk, two systemd services, the ntfy DNS record and the ntfy HTTP check). ntfy went with it - it ran nowhere else - so services/ntfy is deleted, subdomains.ntfy and ntfy_topic are gone from group_vars, and the ntfy playbook is out of site.yml. ntfy_topic already had no readers: the three infra/4xx plays that used it were deleted when their checks were superseded. Two things this exposed. services/ntfy/deploy_ntfy_playbook.yml was pointing at the WRONG MACHINE. It said `hosts: observability`, which resolves to the host `monitoring` (64.226.70.190) - but ntfy ran on watchtower, and ntfy.contrapeso.xyz pointed there. Running it would have installed ntfy on the new VPS. Moot now, but it is the same stale-identity failure as the rest: the group meant watchtower when the play was written, and nobody revisited it when the group changed. Watchtower was in [vps] and NO role group at all, while running caddy, ntfy and Uptime Kuma - nothing in the repo managed any of it. More seriously: ntfy-emergency-app on vipy (avisame.contrapeso.xyz) sends its notifications to https://ntfy.contrapeso.xyz, topic "emergencia". Destroying watchtower breaks it, and it is an EMERGENCY notifier - it would fail silently at exactly the moment it matters. That is NOT resolved here, deliberately: standing ntfy up elsewhere, pointing at ntfy.sh, or retiring the app are all decisions, not cleanups. What this change does is make the break impossible to miss. The URL was derived from subdomains.ntfy, so deleting that would have turned it into an undefined variable buried in a template. It is now an explicit ntfy_service_url in the app's own vars, still holding the old value, with the three options written above it. The ntfy credentials stay in the vault because that app still needs them - the vault was restored from HEAD and only watchtower's push token removed, rather than re-handling the plaintext. Verified: no reference to watchtower or its IP anywhere in the repo; Gatus down from 91 to 85 endpoints, 85 UP, 0 DOWN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 11:26:47 +02:00 · 2026-09-14 11:26:47 +02:00 · 85040d5f67
commit 85040d5f67
parent bf3d21fef7
10 changed files with 171 additions and 282 deletions
--- a/ansible/group_vars/all/main.yml
+++ b/ansible/group_vars/all/main.yml
@ -26,7 +26,6 @@ backup_pull_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfIixKMhA9z+Nvyx6T
 subdomains:
  # Monitoring
  gatus: status
  ntfy: ntfy
  # VPN infrastructure (spacey)
  headscale: headscale
@ -49,9 +48,7 @@ subdomains:
  # DATUM Gateway dashboard (knots-box, proxied via vipy)
  datum_gateway: datum
-# Read by plays targeting managed, monitoring, vpn_control and edge - no one
+# Read by plays across several groups, so global rather than group_vars/<group>.
 # group covers them, so these are global rather than group_vars/<group>.
 ntfy_topic: alerts
 headscale_namespace: counter-net
 # ─────────────────────────────────────────────────────────────────────────────
--- a/ansible/group_vars/all/vault.yml
+++ b/ansible/group_vars/all/vault.yml
@ -1,157 +1,154 @@
 $ANSIBLE_VAULT;1.1;AES256
-30373030336530356533636231303630303132666434393339663833366639366234313434333635
+38396132363836373161663662323635613764643231316132333265396363353833383839613233
-3763366235343633303836383563333734643766346336390a383732313137623263343836616531
+3564323734323561366536303932636435373339363233650a343964613465353137396663666363
-35656262386162346434343462396166653435633330613366633263356638373061643833386336
+63646331366162336537333035386263623264383566646334613332376561613231643235313565
-6438373137326563380a353962396135393365376133626538303862396236363330636564323538
+3638326434613462640a653364646361626636376464633734343332393030393232616362633532
-33616565363066633234633330386338316133613931613931653963356562323336396437313130
+61313161643939383930303362363966326161313237346565323532626135643361633936313135
-33316331333166393161376635306237363033633535613761373762393761363165623462336466
+38396534616239383735353532613834373564336537323531303533323434336166326263376238
-63663265376532663262323463643462336136373639316565396436653966336461656164316135
+35303032336237666235396562313738363130326633336338306632353161336366346332393465
-64303235386236316136623063653133363961303733376166363462303565393761636632666230
+61396163366432323335393362613539313661333431656465336662323336393330373332383834
-66333263626537343935383765303565626237346235356338373063633133326465653133366362
+39636230313566663630656531323530653130643362363562326465643665326335323036346237
-62326563666531343166663563393932393938363663393132616561336363643037363735393233
+32303439356432316564313730653332323731343235336535373462326162656638383735396135
-38336233346138356262616533643835396230663563623237616461626265666339613161616331
+36633264613136323866643162353161306334666463303565333936653861643635336162623831
-34353263363065646336663538663561373536656364623061633863643137633230643931333662
+65386231313465363864623035623364303039323961363237393637303133386530626266663435
-34386433326661336362636466386566636561623339383438626530333431313039623037616430
+37336662643632666164386264646330363135623565306433646437333535363839663330663533
-62333832363531323363646238363733316237646133343337363434373465353463363935323233
+64373231386231643863303933326132646364333339326431646531643962356663326363643734
-37326335343333636337626337643763323336623238636463623933646663356165363062383835
+66346239663438363666613636336430336465623866643861666564353739623238376135346431
-30636630303534636436326530313030626565393439643238353433396638363133353865643135
+38346234333364646230616632646561643638656535306630326332626136396538333534653061
-39666464323537383435636636303930396338333332363264306664383538666236616436353939
+33643562326436636664343339643834323466313131323462653234303834373635313831353538
-61316564623964613839343533316438393337616363633033636231313937366139373031616236
+35333739346563646163333937643232343866383337653462653530666165643361303332363364
-61373763303231376437313332343438376132633361353661336232333966313338373262383663
+36326333333234343761386234666534323430323734616266326231386136336463636231346237
-36363931376132333763363332313534616133326437303637343139383861353138623465623736
+33356361333562656666393630323530333731613964643966383065663635613734383066333532
-62643062396536633730663130353730356637353533616635343166643636616163343832363866
+31366533373533656666666336363666313839316133333062356432633038366363623935373566
-65646266383062373635306239323565383039646334346535653962316434393365646461653366
+33356563663933653539386263663530363239613461383831646262353165343638326663383364
-37396163396362313535393138663833653435353934623432616538663565353165363732316662
+37386330356539343131353633633233323738396339336230666430363965353536343031646163
-35316332363030363963666136653437316639303766376431333738343061663835306339336337
+38306162643733326563373835396264343433666335366136353439356333646131373764336438
-64346137313763656531306636333030323162646533633735646435613566616532363066353361
+61393836303038306132383363633838333665343665303136316131616335643337313362306661
-32326439633432656633303732626361636238323735636631633732383635656436373435343931
+37373137653830636635366266323538393338333463333963343931343433323437663661313366
-64356437326633326163306430306432326134373366616364656361383133656531653333633164
+36616164346265626637616665663461336163343666623162663637633037666564363533623833
-37653934643061343237376664643331623633653437336534313261643765303134626536393938
+35386537306366636533633930326339343136303839306539393866306462303632363163313535
-33393161363539313130376132313864666539363865393035626463313539376635393135633565
+64326530323866313539623661636266663933343165653662373331663334376333393834303938
-35306435636363613735343136663939663435653135376337643034396634623135393039366265
+30373233323664663935343364363731636334343065643662663266343366353031336439383766
-36326332386564303233623863363236356535323538616237303262613261363964643839363837
+39313862616230346661313632613662613335666234333432336262663033633437383565383336
-63336536613139333939343964323835626433373464343730353864303532366535343636663362
+35316365643030616331643333356361373236643731643733386237666430366566303933326563
-34333935316630323137366130356135643962373532346334333931366139356434316431393234
+34336662623431333539616337303739373035313165396535393435383235313633616632616261
-61386632353434323432306130386239663931306538323335333231366234383132623936343465
+37383633306235633939326339303861346335313361316438373630383863643864376462373464
-32393765316136663862663334616234326165336364316137396234343532643639366537343731
+64663631656165316664633336333634643939333464303266366438633039653038356439393565
-33383938363530323337333838643835653238376532376431393439636266306634386233383937
+35343237303533363032396365346232616131306666353766386433393737626639656139373332
-64623230353533643362633563663537343732646261353763366363343231376562336565333362
+65346336373936393866666236346262623666666435363065386563613431626537636364396238
-65623665326136383764373761626461653238303335616463636333383235393037323530636630
+64373133653132366538626262313434353930366338383866653139376538666136383363333666
-65333036653062386332653433366532383132376362356264373432653564316530656163396161
+39636538323437616633356337626130356164363663333637333232663435303537336233643135
-34386133326133393638653961383032323830366232346134323663653436383034356334643137
+64613638643564653431386634376538643564333438343835376633626230303264316337376434
-37663062396534616563363166333535336466373335613733323864363736633263646438336662
+66316231386533313364343131316162623334373065626533303239383439326134343830376438
-39643333366565383130346430623637666132656330653833393836313566393461656638653938
+34333831333035383830386436643239383138633166303033363563313562613564353538373536
-32623664363835376366353532646564643235356637363432363164353837643066643932383163
+33343962346439616439366463383236373437613132326162323437343231666231396332623038
-34643232623734613738613638306664326333346439383830326662653934636536386166323239
+37643932323538633365393033386462663735333034326332663461636363613932346637623638
-63626264613661336663316636353436643262353733303666663866346431353533353435346461
+37323534613439373865646139656462303266373636356630306431323065656533663838303765
-34343763316364303061353030626431613434383062363737356131366234626137656331383433
+30306534343965366566663337313263643832633861336334313835643738656334653135626563
-61653339616336376562326638343462343435613632656632353532376438623462656266326539
+64353062343561626665613162346236393731623533663431333866613932653331613361333039
-35336465366133386534343466323936663934306363353462356530323531383961633336393234
+65336337666264386334373039343965323065626334353030336336396136633363323339313031
-37366266343831386462626539306637623165363066616164393635373631393737613830363661
+62633932613934623065306638656135393438383361376465653431316261633633353031323066
-34656530336634336531373830613039616231343238653637623532386538626365386364353930
+32323466323861316338343137306333326137376434363735393664353761303265653165316634
-32396535643435306231386636363562623239393461393237303030353361633639626632653562
+66623866373730646138333162663830363063326231653636643633623936313164363065396661
-31663261653663356266376264623762353261613163303430623738343939336438333532353436
+64336263333234356263636630666264393734343732613231333331616230333662356164393539
-62343465353033343034373837376538323239653064623135613766343463373436663236616366
+37616533356566323936386331623133666335613765303732313439376461363837623136386638
-30313062663861646331316362653230653139623937343538346230613832623965366666323365
+38313666633436316336396337643666623735396562326538616564363066356136306632313138
-35623635646263396436346162333835343335623364393037353366343537336462636163303761
+36643333346433366233636463626538313465666334373561666433396137323532366662393363
-37346663306339333264643034393630613833343163306430396333656637316662616439343136
+62313434346636386566656437313130663836306265326463383264646564643538333738313436
-35336366306535333733633465343538383462616631326433666663303638633837313065656666
+65343531356138353632346434616134316261306162636439666666326439373836333761333332
-34326136363433303264353331393133626639343166303364343065333266646439663463353833
+30323836633438383166633364366365363561333731306435363433623333316665646334623061
-62346637363265643736616238663033666563326462633562643530383862616265306439376439
+34396131666438613938343364663237393538313632356166346432346530386131623435643566
-63353432343534303433353138663461303165313433353866303838346338363761376333366138
+30613233313739323562373332343537323665623330353135373735663035376138383931333938
-34353035356132383836313134643363636532353834323438313933346534333238646532326263
+30363066303932643465633534663863336339356431393335343261333235396130616165636538
-30663964326564623164306664323265383134616165353634653261383931326663356137363636
+32643335373137373638636537373635616133646333653230376633656561323533376136376564
-36383663666537373863623532376165343334633031626661306664663139616637636162373838
+61306563653736656364383935326635363661636663626530313938646130356132373037373135
-32663863393765343431613536386133653731363266663566303665316435363830303035656230
+64373034656232623530386435633838363262376464376361323161343163373561393839336165
-31366139636332336531326131346237653732313337343336393463373438373531653262363035
+34323137363837616466326364376336646636396364393464626565623934636231643161343032
-39613739323832353332643030343861373836653064326139343965613563363166363236343733
+61616663626130313764393735633866393566633533343737633035623138613662313335656136
-39653666333438613131396566623237643765393435313163333031383461303366396439353437
+61303132333935613236336164396330383136303164633033356638643835313065393836326236
-31376562323231386133356231343366623363396437323866326234613034353664323163646631
+36356235623838656534363639343261623939633864663631343363663262396231343562396530
-36386436653465306665646130616563646564343764346532363961663762303132336137356231
+30653165663332626434636231303637356538666337643361653338303330633932643763323034
-61376565333036313033633732616430343266356435376434386266306432633633316361656538
+66366538303939323232383631383562613732313065323933343761656535366134323164396432
-66393266383735336632653834373561636261663738333039653934636434343165353065353566
+65663339366631623738323135633965623966613433376132363930383061613434386537373332
-34616431616430306234313032346235383065633734323632643065613634343834386664313336
+39623136646638393835646163656364316665646464356535343839393966363631383734633536
-33333861306362333032346633636630326562323430323863373532613831313931303164303438
+62393832346632323330356365393133346134653837663165353834313931356236373664656234
-36353831623262663030313564616634376235383666656434373261643464363264653831386534
+39363431646239323665386639646165303836303731383063663266373236343436633163636535
-63663931643733313239323138393936373435303263663662613266306162373363353334336430
+31393537653962303236383930613339613232643864356230616664313038396266333635396436
-62393835363263666136646631653738393432346331353538306666663864323233386665306332
+36373930646435333863626333333439626663663930663730323236666233386231303730653132
-39353839313731313832333139613830396539306133363236653138643161323337356534643037
+36613763303739376135653531326637313437623338343232303464303639633662373138326134
-38626636353465636166313965346364663238363032383537306532303332643839346230666564
+33616632373031303139313439383465336565323837656632336532373737396230613837376564
-66613261396633346365633730633738306438326234386438373234393537353635626664616239
+63663334363636666534616265623062366132376234326132393732336563313165386334626264
-35363737363537326132376638373839326139643135333234333764626162343766623531643265
+34373934323930663832623338393964353964316132376262353662643534303436643537646361
-32656464383064626566653332323366306138373537663634333833613932666264306164666435
+31663931613164646435656462613433356366633862636461633938343039353531633364373232
-39623335323163636135313361376231316231626335343765323261653134636439643561663463
+31623131616534653533656566613038346133343266373637346663373739656161343765666636
-33376130356362316334386162306333333038316664636464653463313835356461386464653035
+61656466636563643464616565643837623238636466316538636565626261313165346238306261
-38626230376339663361306161386332656230353737376133306464626466653038653266646139
+62346264313739666338376439666633363033336435323932613664373234623931323962393035
-62313937383262633938633735393765616365646165396436653434303835666162373164333938
+33313639626438656134363565396361613965376662386139366163383238346630623064343634
-61643432343237386133316433613030633638633731323936393562376139353033643562633835
+38633962323232666636303830663538333737376437393465626338643632396238353863646263
-36393837383261663761393763613039623263643266303637613461626462313162613762383535
+61363764346330663866396337626237653630633335363436613863613231356262383039613831
-33373064646533366136376563383663633331373161646534653330653566303332616262386564
+62356465626564386537373134663536396231326136376234393966303065333264376630316633
-30313330346463643032363464346537613430633163306365313866373031393965383134323031
+66303262383237386632656231663835383534326333633862383033356263306436623966383161
-63653238343338613033633336366235623332646239613235643637313666616263313833366362
+65663530303836653235616466333736636463323065626335356333373334376337653637666133
-66666633393232303639363966306230663731353730333264663732386235653633303336316139
+30363232383533646331323030303434633162643733623339346132666564396663343131303233
-33646263303164613437343735616362636364376261323364616131383433633864323132353063
+62373431633635356235396130393330643463663239616365663930366138393930343334343432
-32646530323838633962613834663564326137663466623935323665343132653037386339613961
+31616431666361353636316163313430633730366462616332333830353937616466343265343165
-61666363323763636361396330663537386630306637663638343565653437623966313738306366
+36366265626363373164373463353561323437303936393337336238393137633931323537653666
-37633436373434396231396461656634316638393336653964396266363532343864346264323830
+66346231363636326636333736383637376535393961303731386535386138633462323030346330
-32616461353630353730393639306231363662663034626434316661653936636262393761633934
+64353936383162356433306164616264366164623731373733626261653238306362623264666633
-33393863383461316532373839663432336666356332393562393634386533346262306331643733
+64316366393765356632323966343634653235636162353636306337383864326264343434613961
-62373162643763616637316233636163636363363639303662313630336233353162343437356533
+65656433323164363738313535636637383365636435343933333265653333326536363864303866
-30303339666136343665313038643930383539313035373566336139643766353534386232343330
+38396263626166316537326534363637393132363334616134653036663830343764303737383663
-32633836356435326332666266393237353162646537376564656236353138386531323032626636
+31616639373330363365336231333938303138623165666531336336353133653263313263343231
-37373963666666346462343938656466343766323434373064363736386134656232636631653535
+39666639383931653231373535396365386637643738346438333961393138633961376134303664
-38643739333230313265643132356430393035656662356465623836323038663237383439396137
+39336331646236396634356337363336316537643133653561363937373963636238323837303833
-33383335356433383030393131666233663832643638646537363665396435323261613732303363
+33396164656434666239393163363638373364383839633364393137313765613531626337316532
-63653634666164636632306462346166623237303936326435326431636630323562613233386462
+33386465613034663630323638653839666264373235663364643461333238376138303765643364
-64376562316362653937646266366232653733663764633738303162616538656661326261643139
+63623764363830356264306363613266393930376564616438636438373063323030633032326333
-30376233633835333637356134626266376566393962353639663039323439616639316465353038
+66316235643036376165323035386164633536396365326333346638346133336235616132663532
-32373136643162633836663937373664306665643638366237653631313631393934643135393261
+38396636376539656332613963646264316637616235316131316232623131386662333164333534
-35353338393761383431326631633166653932313335346165653364316431333831306135643339
+31366534303166383062316263393833326165346333623830653266646637663364373365626335
-63316538343963336237653366343836343330656364373661343866656235303931633432313966
+61623837323934343131353336393163303831613832303530356639393465633461326566363138
-35393631333263363533316537616132323038303230653838326266336434336532346439626537
+39363730353862383365386339353536366537363933663566306132626135336339646533323763
-30656334316239356261386633393635643563336230326362303034633235323830343435646136
+65616664393166666461643038336232303934306338323331366233643061373532623163663931
-63333237346164323763633665636234313263393366636233393538353335383933636331656261
+37343839346637376337613438616131633239623234613539323138353233343238653661626462
-39313733383238613634363139326536616237353031666232663161653763326162386464626533
+34353666643230633562323631303732373332616335326131396131653038643531646562646431
-39633931353035356561643564656637633163396561356430663636313231333962363765373536
+33626130343834396439323138393162363831393534383839663638663838633061666534336138
-62613366616538306331623537613564353437366239616332626237623466613931653339333431
+35313234653565316437336439376563666138653661613062643262656233316262343736363235
-39646239383438623063663063333861356536326637656337303561346432333539656431373733
+32623466626366343965373638323239393933386434633531303363313061313832663166653634
-31316461343636313430303434336535353430353932373536366162333938646464653763356237
+39333835303066303231363663666461323434363237646133323661386663633939313965363864
-39363938373065663732393862383031336438396164366664323130343734323130306662343138
+38303861383565303766306533633132376632373234646338306264343065633631303935313266
-35383734663465303034393561306264316139623265386134323162373734393364393230333234
+30333836616164363735363934386165663839393731633039383538623833353632373636656436
-31383336363832373962613535346136353037383065363066653935373435383037313764636266
+62343934363834613464626364303434393262373365643130373437393730393964316130613565
-65383133333433326662626639356162303261393866373732366665353465356534333431633639
+35373938306436666364646136306166663132393736383862623635316265386361663333356431
-39303539346266316132643537393564346562353533363438376363613036613739633939353564
+35336363623535336631366531316661333130393034383566613138653834383333623437356139
-37313764316661663939396664376538626437366534383630613030323465396535366537346263
+36353434626531616361333138386338636132646533326565336534643632316363313732336233
-31636338353233306333653962373763353439346337323839633238636336653439386362333533
+38656134383565303132643161393032333334316265343862666532303438653161626665326632
-31653264343435633432646633386531386139366461646133396334326631333632393462666232
+31343435393136313865356166363165613730373633623963303633343165613230393430633831
-35383735356232336335336435343932613433393530653633616466373965383935383861366136
+31376361656361666434343866353037333834353735396566643935393439343334376461323133
-31303936616333396638326465633531373164626232356362613130376434303633366238336666
+61663835656137356665623265376664336332313432636664313735643739626333383566336331
-37643035303932393265363037646430343866323163386234633739393938656530646533633163
+35306539386433633636356636383734636633666137653334333162386564643464306538366461
-61653135633436653566303832383564663462353235353134313033616364636561643663383835
+33323130656331313834323839303736366436633634356261363335343231303331353339313031
-30326561653934636164636363363736346630316263616664386233373535356339336239653333
+61613563383937623162616133313633653833666634643566643766336533303566383561633237
-32336362363033616337663936623331663134656665643739626637353739666231643766346134
+61316562353933333666323037376334326430623035643630323565386532313530383835343265
-63353631663631643436633935306261373939323530363639666366373531626231366138643766
+61616165396664616332313262666534396537356564656236313864326533306338373362613034
-34623137643233333232303339323464333566633038333539356337306134383330353632383965
+38623162336138366262626538366333373830623634616261353764666563663138663862656339
-34626461316261363830376664343336666132376365333635666131653933323562666635353763
+65346137646263393064393536613963666434373663636336373531316464336264313061316439
-66376335663861623332623436333161353161336337313136613531646632363531383138343130
+39623832666536356334383261653431343837663163623035366162396465646239613130643837
-39306330336239393965383939333765633532376539653661353965373030313038666161313265
+64393735333438376262323834313330363434366163653833376334666666653535633330636535
-37653164343832333937356434646230623736646561623561643135626531343063626636663333
+66346665643332393235313062313164333030353265386632326431333036333861356231666531
-30363937613834663434313038653739613933656534626532326136323336386164366339313830
+63323966623361343363623561386633363734323864623961393832653037353766326637663735
-35396130386634666363656237373835313563343961633838383766663933376439326364616530
+63396565326632303366396539396435626235643833323864663731396636313237373832646566
-64383136623739623137363562666431333565643130346166663531373738643761363337333536
+66333839643734303734376338353037393266353232316264353736346338646262386636616335
-38313866363339393066643339653734316266393037396432356138303137343536313134623434
+38313838363064643739633065343535643762626165343862646362613462386532356166383437
-62623362383863303539663264386430353436613261643865666534356538626538623630643766
+38646130313462393863376339336436336630643061363130663831313532663466363936343934
-66646266336238396230613838626161336637313564303465663034653232306363306430633438
+31656230323564303638653832626364633631336336613663356562636436643336323238393436
-35303364356531636565303965323539393233663566646562383863323461396166623032396338
+63343462366436653365383766366262373332313339626431326538333562346438303730343937
-36346537343633643864323163623464623539616434376137313538646333666339626437656330
+33326365636631633365336332636638353663303633623137643764303930656336323963643835
-31326463666335626361616566383065363264356663323435646337663566636436356336376531
+34376136353532613938326432663835363530653264383561353533663863376661636137336437
-38636466386539663937326462636464613638653833666263333134396236313432643030366138
+61353365333661623038363535643035393335613632656330333938623335343436663763646362
-66326364623435356633666463333066373238363864356230633634353764616331303163636266
+63633130373530386235663661613335323531363234343465313861656639646461613936336134
-36306239323561326134393665643031663362636535656139376237383130616663316362326366
+30613937376666623761626438396635313563306564633433613330633664356636636565613166
-61623236356562346233643332663261626333646638373262613664623935626266306230356164
+32633430646139383362623133346435633063633466656635333034343937643639346536393134
-32336363383538333935333062386366653839303139366565343933346266623637363534393335
+39383664383234643265323563646465373636303261646331366439666466343636333330633131
-31346431313631373535306138633930383536613636393034343434623664376237373066633036
+37363963373361613331663263623737373464663831326630663432316336336163343866663834
-30323864613430333239326366613439613632633933656631356237333930303330326264646563
+6630
 38303066323563303262663530326637663964373039336431353339626436363335326466313262
 35386533633833363038636339663265626261373464633565666166633135333461363165353931
 63633332613966663133
--- a/ansible/host_vars/watchtower/main.yml
+++ b/ansible/host_vars/watchtower/main.yml
@ -1,13 +0,0 @@
 ---
 # Systemd services deployed on this host, monitored every 5 minutes.
 #
 # The fact lives with the machine rather than in a central map, for the same
 # reason the cross-host ports do: "what runs here" is a property of the host,
 # and a central list is one more thing to forget to update when a service moves.
 #
 # Only units WE deploy belong here. Distro units (ssh, cron) have their own
 # supervision and would be noise.
 monitored_services:
  - caddy
  - ntfy
--- a/ansible/infra/402_public_monitoring.yml
+++ b/ansible/infra/402_public_monitoring.yml
@ -22,7 +22,6 @@
    # ansible_host - if a box is renumbered, inventory is the one edit.
    dns_records:
      - {sub: "{{ subdomains.gatus }}",              host: monitoring}
      - {sub: "{{ subdomains.ntfy }}",               host: watchtower}
      - {sub: "{{ subdomains.headscale }}",          host: spacey}
      - {sub: "{{ subdomains.vaultwarden }}",        host: vipy}
      - {sub: "{{ subdomains.forgejo }}",            host: vipy}
@ -42,7 +41,6 @@
    # there would go green precisely when the auth broke.
    public_sites:
      - {name: gatus,       sub: "{{ subdomains.gatus }}",              path: "/",       status: 401}
      - {name: ntfy,        sub: "{{ subdomains.ntfy }}",               path: "/",       status: 200}
      - {name: headscale,   sub: "{{ subdomains.headscale }}",          path: "/health", status: 200}
      - {name: vaultwarden, sub: "{{ subdomains.vaultwarden }}",        path: "/",       status: 200}
      - {name: forgejo,     sub: "{{ subdomains.forgejo }}",            path: "/",       status: 200}
--- a/ansible/inventory.ini
+++ b/ansible/inventory.ini
@ -1,6 +1,5 @@
 [vps]
 vipy ansible_host=167.172.107.33 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 watchtower ansible_host=164.92.239.72 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 spacey ansible_host=64.227.112.128 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
 monitoring ansible_host=64.226.70.190 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
--- a/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml
+++ b/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml
@ -6,8 +6,17 @@
  vars:
    ntfy_emergency_app_subdomain: "{{ subdomains.ntfy_emergency_app }}"
    ntfy_emergency_app_domain: "{{ ntfy_emergency_app_subdomain }}.{{ root_domain }}"
-    ntfy_service_domain: "{{ subdomains.ntfy }}.{{ root_domain }}"
+    # ⚠ UNRESOLVED: this app sends its notifications to an ntfy server, and the
-    ntfy_emergency_app_ntfy_url: "https://{{ ntfy_service_domain }}"
+    # server it points at ran on watchtower, which is being destroyed. This was
    # derived from subdomains.ntfy, which is now gone with it.
    #
    # Until an ntfy server exists again this URL is dead, and the app fails
    # silently at exactly the moment it matters - it is an EMERGENCY notifier.
    # Three ways out, none of them automatic:
    #   * stand ntfy up somewhere else (the monitoring VPS, or vipy)
    #   * point this at the public ntfy.sh
    #   * retire the app
    ntfy_emergency_app_ntfy_url: "{{ ntfy_service_url }}"
    ntfy_emergency_app_ntfy_user: "{{ ntfy_username | default('') }}"
    ntfy_emergency_app_ntfy_password: "{{ ntfy_password | default('') }}"
--- a/ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml
+++ b/ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml
@ -14,3 +14,9 @@ remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_h
 remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
 remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
 remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
 # Where the emergency notifications are sent. This pointed at ntfy.contrapeso.xyz
 # on watchtower; that host is being destroyed, so this MUST be repointed before
 # the app can work again. Left at the old value so the break is visible rather
 # than silently defaulted to something plausible.
 ntfy_service_url: "https://ntfy.contrapeso.xyz"
--- a/ansible/services/ntfy/deploy_ntfy_playbook.yml
+++ b/ansible/services/ntfy/deploy_ntfy_playbook.yml
@ -1,97 +0,0 @@
 - name: Deploy ntfy and configure Caddy reverse proxy
  hosts: observability
  become: yes
  vars_files:
    - ./ntfy_vars.yml
  vars:
    ntfy_subdomain: "{{ subdomains.ntfy }}"
    ntfy_domain: "{{ ntfy_subdomain }}.{{ root_domain }}"
  tasks:
    - name: Ensure /etc/apt/keyrings exists
      file:
        path: /etc/apt/keyrings
        state: directory
        mode: '0755'
    - name: Download and dearmor ntfy GPG key
      shell: curl -fsSL https://archive.heckel.io/apt/pubkey.txt | gpg --dearmor -o /etc/apt/keyrings/archive.heckel.io.gpg
      args:
        creates: /etc/apt/keyrings/archive.heckel.io.gpg
    - name: Add ntfy APT repository
      copy:
        dest: /etc/apt/sources.list.d/archive.heckel.io.list
        content: |
          deb [arch=amd64 signed-by=/etc/apt/keyrings/archive.heckel.io.gpg] https://archive.heckel.io/apt debian main
        mode: '0644'
    - name: Update APT cache
      apt:
        update_cache: yes
    - name: Install ntfy
      apt:
        name: ntfy
        state: present
    - name: Ensure ntfy cache directories exist
      file:
        path: "{{ item }}"
        state: directory
        owner: ntfy
        group: ntfy
        mode: '0755'
      loop:
        - /var/cache/ntfy
        - /var/cache/ntfy/attachments
    - name: Deploy ntfy configuration file
      copy:
        dest: /etc/ntfy/server.yml
        content: |
          base-url: "http://{{ ntfy_domain }}"
          listen-http: ":{{ ntfy_port }}"
          cache-file: "/var/cache/ntfy/cache.db"
          attachment-cache-dir: "/var/cache/ntfy/attachments"
          behind-proxy: true
          auth-file: "/var/lib/ntfy/user.db"
          auth-default-access: "deny-all"
        owner: root
        group: root
        mode: '0644'
      notify: Restart ntfy
    - name: Enable and start ntfy service
      systemd:
        name: ntfy
        enabled: yes
        state: started
    - name: Create ntfy admin user
      shell: |
        (echo "{{ ntfy_password }}"; echo "{{ ntfy_password }}") | ntfy user add --role=admin "{{ ntfy_username }}"
    - name: Publish ntfy through Caddy
      ansible.builtin.include_role:
        name: caddy_site
      vars:
        caddy_site_name: ntfy
        caddy_site_domain: "{{ ntfy_domain }}, http://{{ ntfy_domain }}"
        # Raw body: ntfy needs a plain-HTTP listener for its CLI/app clients,
        # with only GETs to the docs and topic paths redirected to HTTPS.
        caddy_site_body: |
          reverse_proxy 127.0.0.1:{{ ntfy_port }}
          @httpget {
              protocol http
              method GET
              path_regexp ^/([-_a-z0-9]{0,64}$|docs/|static/)
          }
          redir @httpget https://{host}{uri}
  handlers:
    - name: Restart ntfy
      systemd:
        name: ntfy
        state: restarted
--- a/ansible/services/ntfy/ntfy_vars.yml
+++ b/ansible/services/ntfy/ntfy_vars.yml
@ -1,3 +0,0 @@
 ntfy_port: 6674
 # ntfy_topic lives in group_vars/all/main.yml
--- a/ansible/site.yml
+++ b/ansible/site.yml
@ -56,7 +56,6 @@
 - import_playbook: services/vaultwarden/deploy_vaultwarden_playbook.yml
 - import_playbook: services/forgejo/deploy_forgejo_playbook.yml
 - import_playbook: services/lnbits/deploy_lnbits_playbook.yml
 - import_playbook: services/ntfy/deploy_ntfy_playbook.yml
 - import_playbook: services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml
 - import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml
@ -80,9 +79,6 @@
 #   infra/nodito/30_proxmox_bootstrap   one-shot: bare-metal bootstrap, run once
 #   infra/nodito/33_..._cloud_template  one-shot: builds the VM template
 #
 #   services/ntfy/setup_ntfy_uptime_    creates a notification channel INSIDE
 #     kuma_notification.yml             Uptime Kuma. Kuma-specific tooling, not
 #                                       a deployment, and Kuma is being retired.
 #
 #   services/vaultwarden/disable_       deliberate manual actions, not convergence
 #     vaultwarden_sign_ups_playbook.yml
		`@ -1,3 +0,0 @@`
			`ntfy_port: 6674`

			`# ntfy_topic lives in group_vars/all/main.yml`