stuff

.gitignore

@@ -1,3 +1,21 @@
+# OpenTofu / Terraform
+.terraform/
+.tofu/
+.terraform.lock.hcl
+.tofu.lock.hcl
+terraform.tfstate
+terraform.tfstate.*
+crash.log
+*.tfvars
+*.tfvars.json
+
+test-inventory.ini
 inventory.ini
 venv/*
-.env
+.env
+
+# Secrets and sensitive files
+*_secrets.yml
+*_secrets.yaml
+secrets/
+.secrets/

01_infra_setup.md

@@ -18,23 +18,26 @@ This describes how to prepare each machine before deploying services on them.
 * Getting and configuring the domain is outside the scope of this repo. Whenever a service needs you to set up a subdomain, it will be mentioned explictly.
 * You should add the domain to the var `root_domain` in `ansible/infra_vars.yml`.
 
-## Prepare the VPSs (vipy and watchtower)
+## Prepare the VPSs (vipy, watchtower and spacey)
 
 ### Source the VPSs
 
 * The guide is agnostic to which provider you pick, but has been tested with VMs from https://99stack.com and contains some operations that are specifically relevant to their VPSs.
 * The expectations are that the VPS ticks the following boxes:
-  + Runs Debian 12 bookworm.
+  + Runs Debian 12/13 bookworm.
   + Has a public IP4 and starts out with SSH listening on port 22.
   + Boots with one of your SSH keys already authorized. If this is not the case, you'll have to manually drop the pubkey there before using the playbooks.
-* You will need two VPSs: one to host most services, and another tiny one to monitor Uptime. We use two to prevent the monitoring service from falling down with the main machine.
+* You will need three VPSs:
+  + One to host most services, 
+  + Another tiny one to monitor Uptime. We use a different one to prevent the monitoring service from falling down with the main machine.
+  + A final one to run the headscale server, since the main VPS needs to be part of the mesh network and can't do so while also running the coordination server.
 * Move on once your VPSs are running and satisfies the prerequisites.
 
 ### Prepare Ansible vars
 
-* You have an example `ansible/example.inventory.ini`. Copy it with `cp ansible/example.inventory.ini ansible/inventory.ini` and fill in with the values for your VPSs. `[vipy]` is the services VPS. `[watchtower]` is the watchtower VPS.
+* You have an example `ansible/example.inventory.ini`. Copy it with `cp ansible/example.inventory.ini ansible/inventory.ini` and fill in the `[vps]` group with host entries for each machine (`vipy` for services, `watchtower` for uptime monitoring, `spacey` for headscale).
 * A few notes:
-  * The guides assume you'll only have one VPS in the `[vipy]` group. Stuff will break if you have multiple, so avoid that.
+  * The guides assume you'll only have one `vipy` host entry. Stuff will break if you have multiple, so avoid that.
 
 ### Create user and secure VPS access
 
@@ -45,6 +48,127 @@ This describes how to prepare each machine before deploying services on them.
 
 Note that, by applying these playbooks, both the root user and the `counterweight` user will use the same SSH pubkey for auth.
 
+Checklist:
+- [ ] All 3 VPS are accessible with the `counterweight` user
+- [ ] All 3 VPS have UFW up and running
+
+## Prepare Nodito Server
+
+### Source the Nodito Server
+
+* This setup is designed for a local Nodito server running in your home environment.
+* The expectations are that the Nodito server:
+  + Runs Proxmox VE (based on Debian).
+  + Has a predictable local IP address.
+  + Has root user with password authentication enabled (default Proxmox state).
+  + SSH is accessible on port 22.
+
+### Prepare Ansible vars for Nodito
+
+* Ensure your inventory contains a `[nodito_host]` group and the `nodito` host entry (copy the example inventory if needed) and fill in with values.
+
+### Bootstrap SSH Key Access and Create User
+
+* Nodito starts with password authentication enabled and no SSH keys configured. We need to bootstrap SSH key access first.
+* Run the complete setup with: `ansible-playbook -i inventory.ini infra/nodito/30_proxmox_bootstrap_playbook.yml -e 'ansible_user=root'`
+* This single playbook will:
+  * Set up SSH key access for root
+  * Create the counterweight user with SSH keys
+  * Update and secure the system
+  * Disable root login and password authentication
+  * Test the final configuration
+* For all future playbooks targeting nodito, use the default configuration (no overrides needed).
+
+Note that, by applying these playbooks, both the root user and the `counterweight` user will use the same SSH pubkey for auth, but root login will be disabled.
+
+### Switch to Community Repositories
+
+* Proxmox VE installations typically come with enterprise repositories enabled, which require a subscription. To avoid subscription warnings and use the community repositories instead:
+* Run the repository switch with: `ansible-playbook -i inventory.ini infra/nodito/32_proxmox_community_repos_playbook.yml`
+* This playbook will:
+  * Detect whether your Proxmox installation uses modern deb822 format (Proxmox VE 9) or legacy format (Proxmox VE 8)
+  * Remove enterprise repository files and create community repository files
+  * Disable subscription nag messages in both web and mobile interfaces
+  * Update Proxmox packages from the community repository
+  * Verify the changes are working correctly
+* After running this playbook, clear your browser cache or perform a hard reload (Ctrl+Shift+R) before using the Proxmox VE Web UI to avoid UI display issues.
+
+### Deploy Infra Monitoring (Disk, Health, CPU Temp)
+
+* Nodito can run the same monitoring stack used elsewhere: disk usage, heartbeat healthcheck, and CPU temperature alerts feeding Uptime Kuma.
+* Playbooks to run (in any order):
+  * `ansible-playbook -i inventory.ini infra/410_disk_usage_alerts.yml`
+  * `ansible-playbook -i inventory.ini infra/420_system_healthcheck.yml`
+  * `ansible-playbook -i inventory.ini infra/430_cpu_temp_alerts.yml`
+* Each playbook automatically:
+  * Creates/updates the corresponding monitor in Uptime Kuma (including ntfy notification wiring)
+  * Installs any required packages (curl, lm-sensors, jq, bc, etc.)
+  * Creates the monitoring script(s) and log files
+  * Sets up systemd services and timers for automated runs
+  * Sends alerts to Uptime Kuma when thresholds are exceeded or heartbeats stop
+
+### Setup ZFS Storage Pool
+
+* The nodito server can be configured with a ZFS RAID 1 storage pool for Proxmox VM storage, providing redundancy and data integrity.
+* Before running the ZFS pool setup playbook, you need to identify your disk IDs and configure them in the variables file:
+  * SSH into your nodito server and run: `ls -la /dev/disk/by-id/ | grep -E "(ata-|scsi-|nvme-)"`
+  * This will show you the persistent disk identifiers for all your disks. Look for the two disks you want to use for the ZFS pool.
+  * Example output:
+    ```
+    lrwxrwxrwx 1 root root  9 Dec 15 10:30 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1234567 -> ../../sdb
+    lrwxrwxrwx 1 root root  9 Dec 15 10:30 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K7654321 -> ../../sdc
+    ```
+  * Update `ansible/infra/nodito/nodito_vars.yml` with your actual disk IDs:
+    ```yaml
+    zfs_disk_1: "/dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1234567"
+    zfs_disk_2: "/dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K7654321"
+    ```
+* Run the ZFS pool setup with: `ansible-playbook -i inventory.ini infra/nodito/32_zfs_pool_setup_playbook.yml`
+* This will:
+  * Validate Proxmox VE and ZFS installation
+  * Install ZFS utilities and kernel modules
+  * Create a RAID 1 (mirror) ZFS pool named `proxmox-storage` with optimized settings
+  * Configure ZFS pool properties (ashift=12, compression=lz4, atime=off, etc.)
+  * Export and re-import the pool for Proxmox compatibility
+  * Configure Proxmox to use the ZFS pool storage (zfspool type)
+  * Enable ZFS services for automatic pool import on boot
+* **Warning**: This will destroy all data on the specified disks. Make sure you're using the correct disk IDs and that the disks don't contain important data.
+
+### Build Debian Cloud Template for Proxmox
+
+* After storage is ready, create a reusable Debian cloud template so future Proxmox VMs can be cloned in seconds.
+* Run: `ansible-playbook -i inventory.ini infra/nodito/33_proxmox_debian_cloud_template.yml`
+* This playbook:
+  * Downloads the latest Debian generic cloud qcow2 image (override via `debian_cloud_image_url`/`debian_cloud_image_filename`)
+  * Imports it into your Proxmox storage (defaults to the configured ZFS pool) and builds VMID `9001` as a template
+  * Injects your SSH keys, enables qemu-guest-agent, configures DHCP networking, and sizes the disk (default 10 GB)
+  * Drops a cloud-init snippet so clones automatically install qemu-guest-agent and can run upgrades on first boot
+* Once it finishes, provision new machines with `qm clone 9001 <vmid> --name <vmname>` plus your usual cloud-init overrides.
+
+### Provision VMs with OpenTofu
+
+* Prefer a declarative workflow? The `tofu/nodito` project clones VM definitions from the template automatically.
+* Quick start (see `tofu/nodito/README.md` for full details):
+  1. Install OpenTofu, copy `terraform.tfvars.example` to `terraform.tfvars`, and fill in the Proxmox API URL/token plus your SSH public key.
+  2. Define VMs in the `vms` map (name, cores, memory, disk size, `ipconfig0`, optional `vlan_tag`). Disks default to the `proxmox-tank-1` ZFS pool.
+  3. Run `tofu init`, `tofu plan -var-file=terraform.tfvars`, and `tofu apply -var-file=terraform.tfvars`.
+* Each VM is cloned from the `debian-13-cloud-init` template (VMID 9001), attaches to `vmbr0`, and boots with qemu-guest-agent + your keys injected via cloud-init. Updates to the tfvars map let you grow/shrink the fleet with a single `tofu apply`.
+
+## General prep for all machines
+
+### Set up Infrastructure Secrets
+
+* Create `ansible/infra_secrets.yml` based on the example file:
+  ```bash
+  cp ansible/infra_secrets.yml.example ansible/infra_secrets.yml
+  ```
+* Edit `ansible/infra_secrets.yml` and add your Uptime Kuma credentials:
+  ```yaml
+  uptime_kuma_username: "admin"
+  uptime_kuma_password: "your_password"
+  ```
+* **Important**: Never commit this file to version control (it's in `.gitignore`)
+
 ## GPG Keys
 
 Some of the backups are stored encrypted for security. To allow this, fill in the gpg variables listed in `example.inventory.ini` under the `lapy` block.

02_vps_core_services_setup.md

@@ -1,6 +1,6 @@
 # 02 VPS Core Services Setup
 
-Now that Vipy is ready, we need to deploy some basic services which are foundational for the apps we're actually interested in.
+Now that the VPSs are ready, we need to deploy some basic services which are foundational for the apps we're actually interested in.
 
 This assumes you've completed the markdown `01`.
 
@@ -28,6 +28,9 @@ Simply run the playbook:
 ansible-playbook -i inventory.ini infra/910_docker_playbook.yml
 ```
 
+Checklist:
+- [ ] All 3 VPSs responde to `docker version`
+- [ ] All 3 VPSs responde to `docker compose version`
 
 ## Deploy Caddy
 
@@ -40,6 +43,9 @@ ansible-playbook -i inventory.ini infra/910_docker_playbook.yml
 
 * Starting config will be empty. Modifying the caddy config file to add endpoints as we add services is covered by the instructions of each service.
 
+Checklist:
+- [ ] All 3 VPSs have Caddy up and running
+
 
 ## Uptime Kuma
 
@@ -47,9 +53,8 @@ Uptime Kuma gets used to monitor the availability of services, keep track of the
 
 ### Deploy
 
-* Decide what subdomain you want to serve Uptime Kuma on and add it to `services/uptime_kuma/uptime_kuma_vars.yml` on the `uptime_kuma_subdomain`.
+* Decide what subdomain you want to serve Uptime Kuma on and add it to `services/services_config.yml` on the `uptime_kuma` entry.
     * Note that you will have to add a DNS entry to point to the VPS public IP.
-* Make sure docker is available on the host.
 * Run the deployment playbook: `ansible-playbook -i inventory.ini services/uptime_kuma/deploy_uptime_kuma_playbook.yml`.
 
 ### Set up backups to Lapy
@@ -69,6 +74,50 @@ Uptime Kuma gets used to monitor the availability of services, keep track of the
 * Overwrite the data folder with one of the backups.
 * Start it up again.
 
+Checklist:
+- [ ] Uptime kuma is accesible at the FQDN
+- [ ] The backup script runs fine
+- [ ] You have stored the credentials of the Uptime kuma admin user
+
+
+## ntfy
+
+ntfy is a notifications server.
+
+### Deploy
+
+* Decide what subdomain you want to serve ntfy on and add it to `services/ntfy/ntfy_vars.yml` on the `ntfy_subdomain`.
+    * Note that you will have to add a DNS entry to point to the VPS public IP.
+* Ensure the admin user credentials are set in `ansible/infra_secrets.yml` under `ntfy_username` and `ntfy_password`. This user is the only one authorised to send and read messages from topics.
+* Run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy/deploy_ntfy_playbook.yml`.
+* Run this playbook to create a notifaction entry in uptime kuma that points to your freshly deployed ntfy instance: `ansible-playbook -i inventory.ini services/ntfy/setup_ntfy_uptime_kuma_notification.yml`
+
+### Configure
+
+* You can visit the ntfy web UI at the FQDN you configured.
+* You can start using notify to send alerts with uptime kuma by visiting the uptime kuma UI and using the credentials for the ntfy admin user.
+* To receive alerts on your phone, install the official ntfy app: https://github.com/binwiederhier/ntfy-android.
+* You can also subscribe on the web UI on your laptop.
+
+### Backups
+
+Given that ntfy is almost stateless, no backups are made. If it blows up, simply set it up again.
+
+Checklist
+- [ ] ntfy UI is reachable
+- [ ] You can see the notification in uptime kuma and test it successfully
+
+## VPS monitoring scripts
+
+### Deploy
+
+- Run playbooks:
+    - `ansible-playbook -i inventory.ini infra/410_disk_usage_alerts.yml --limit vps`
+    - `ansible-playbook -i inventory.ini infra/420_system_healthcheck.yml --limit vps`
+
+Checklist:
+- [ ] You can see both the system healthcheck and disk usage check for all VPSs in the uptime kuma UI.
+- [ ] The checks are all green after ~30 min.
 
 ## Vaultwarden
 
@@ -102,6 +151,12 @@ Vaultwarden is a credentials manager.
 * Stop Vaultwarden.
 * Overwrite the data folder with one of the backups.
 * Start it up again.
+* Be careful! The restoring of a backup doesn't include the signup behaviour. If you deployed a new instance and restored a backup, you still need to manually repeat as described above the disabling of the sign ups.
+
+Checklist
+- [ ] The service is reachable at the URL
+- [ ] You have stored the admin creds properly
+- [ ] You can't create another user at the /signup path
 
 ## Forgejo
 
@@ -120,28 +175,27 @@ Forgejo is a git server.
 * You can tweak more settings from that point on.
 * SSH cloning should work out of the box (after you've set up your SSH pub key in Forgejo, that is).
 
+### Set up backups to Lapy
 
-## ntfy
+* Make sure rsync is available on the host and on Lapy.
+* Ensure GPG is configured with a recipient in your inventory (the backup script requires `gpg_recipient` to be set).
+* Run the backup playbook: `ansible-playbook -i inventory.ini services/forgejo/setup_backup_forgejo_to_lapy.yml`.
+* A first backup process gets executed and then a cronjob is set up to refresh backups periodically. The script backs up both the data and config directories. Backups are GPG encrypted for safety. Note that the Forgejo service is stopped during backup to ensure consistency.
 
-ntfy is a notifications server.
+### Restoring to a previous state
 
-### Deploy
+* Stop Forgejo.
+* Decrypt the backup: `gpg --decrypt forgejo-backup-YYYY-MM-DD.tar.gz.gpg | tar -xzf -`
+* Overwrite the data and config directories with the restored backup.
+* Ensure that files in `/var/lib/foregejo/` are owned by the right user.
+* Start Forgejo again.
+* You may need to refresh ssh pub key so your old SSH driven git remotes work. Go to site administration, dashboard, and run task `Update the ".ssh/authorized_keys" file with Forgejo SSH keys.`.
 
-* Decide what subdomain you want to serve ntfy on and add it to `services/ntfy/ntfy_vars.yml` on the `ntfy_subdomain`.
-    * Note that you will have to add a DNS entry to point to the VPS public IP.
-* Before running the playbook, you should decide on a user and password for the admin user. This user is the only one authorised to send and read messages from topics. Once you've picked, export them in your terminal like this `export NTFY_USER=admin; export NTFY_PASSWORD=secret`.
-* In the same shell, run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy/deploy_ntfy_playbook.yml`.
-
-### Configure
-
-* You can visit the ntfy web UI at the FQDN you configured.
-* You can start using notify to send alerts with uptime kuma by visiting the uptime kuma UI and using the credentials for the ntfy admin user.
-* To receive alerts on your phone, install the official ntfy app: https://github.com/binwiederhier/ntfy-android.
-* You can also subscribe on the web UI on your laptop.
-
-### Backups
-
-Given that ntfy is almost stateless, no backups are made. If it blows up, simply set it up again.
+Checklist:
+- [ ] Forgejo is accessible at the FQDN
+- [ ] You have stored the admin credentials properly
+- [ ] The backup script runs fine
+- [ ] SSH cloning works after setting up your SSH pub key
 
 
 ## LNBits
@@ -150,7 +204,7 @@ LNBits is a Lightning Network wallet and accounts system.
 
 ### Deploy
 
-* Decide what subdomain you want to serve LNBits on and add it to `services/lnbits/lnbits_vars.yml` on the `lnbits_subdomain`.
+* Decide what subdomain you want to serve LNBits on and add it to `ansible/services_config.yml` under `lnbits`.
     * Note that you will have to add a DNS entry to point to the VPS public IP.
 * Run the deployment playbook: `ansible-playbook -i inventory.ini services/lnbits/deploy_lnbits_playbook.yml`.
 
@@ -181,49 +235,16 @@ ntfy-emergency-app is a simple web application that allows trusted people to sen
 
 ### Deploy
 
-* Decide what subdomain you want to serve the emergency app on and add it to `services/ntfy-emergency-app/ntfy_emergency_app_vars.yml` on the `ntfy_emergency_app_subdomain`.
+* Decide what subdomain you want to serve the emergency app on and update `ansible/services_config.yml` under `ntfy_emergency_app`.
     * Note that you will have to add a DNS entry to point to the VPS public IP.
 * Configure the ntfy settings in `ntfy_emergency_app_vars.yml`:
     * `ntfy_emergency_app_topic`: The ntfy topic to send messages to (default: "emergency")
-    * `ntfy_emergency_app_ntfy_url`: Your ntfy server URL (default: "https://ntfy.sh")
-    * `ntfy_emergency_app_ntfy_user`: Username for ntfy authentication (optional)
-    * `ntfy_emergency_app_ntfy_password`: Password for ntfy authentication (optional)
     * `ntfy_emergency_app_ui_message`: Custom message displayed in the web interface
+* Ensure `infra_secrets.yml` contains `ntfy_username` and `ntfy_password` with the credentials the app should use.
 * Make sure docker is available on the host.
 * Run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml`.
 
 
-## Personal Blog
-
-Personal blog is a static website served directly by Caddy.
-
-### Deploy
-
-* Decide what subdomain you want to serve the blog on and add it to `services/personal-blog/personal_blog_vars.yml` on the `personal_blog_subdomain`.
-    * Note that you will have to add a DNS entry to point to the VPS public IP.
-* Configure the git repository settings in `personal_blog_vars.yml`:
-    * `personal_blog_git_repo`: The HTTPS URL to your git repository (default: "https://forgejo.contrapeso.xyz/counterweight/pablohere.git")
-    * `personal_blog_source_folder`: The folder within the repo containing static files (default: "public")
-* Set up a Forgejo deploy token:
-    * Go to your repository → Settings → Deploy Tokens
-    * Create a new token with "Read" permissions
-    * Copy the token (you won't see it again)
-* Export the token as an environment variable: `export PERSONAL_BLOG_DEPLOY_TOKEN=your_token_here`
-* Run the deployment playbook: `ansible-playbook -i inventory.ini services/personal-blog/deploy_personal_blog_playbook.yml`.
-
-### Configure
-
-* The blog will be automatically updated every hour via a cron job that pulls the latest changes from the git repository.
-* Static files are served directly by Caddy from the configured webroot directory.
-* No additional configuration is needed - the site will be available at your configured domain.
-
-### Updating content
-
-* Simply push changes to the `master` branch of your git repository.
-* The cron job will automatically pull and deploy updates within an hour.
-* For immediate updates, you can manually run: `/usr/local/bin/update-personal-blog.sh` on the server.
-
-
 ## Headscale
 
 Headscale is a self-hosted Tailscale control server that allows you to create your own Tailscale network.
@@ -237,20 +258,30 @@ Headscale is a self-hosted Tailscale control server that allows you to create yo
 ### Configure
 
 * **Network Security**: The network starts with a deny-all policy - no devices can communicate with each other until you explicitly configure ACL rules in `/etc/headscale/acl.json`.
-* After deployment, you need to create a namespace and generate pre-auth keys for your devices.
-* SSH into your VPS and run the following commands:
-    ```bash
-    # Create a namespace
-    headscale user create counter-net
-    
-    # Generate a pre-auth key for device registration
-    headscale preauthkeys create --user 1 # Assumes you've only created one user
-    ```
-* Copy the generated pre-auth key - you'll need it to register your devices.
+* After deployment, the namespace specified in `services/headscale/headscale_vars.yml` is automatically created.
 
 ### Connect devices
 
+#### Automated method (for servers reachable via SSH from lapy)
+
+* Use the Ansible playbook to automatically join machines to the mesh:
+    ```bash
+    ansible-playbook -i inventory.ini infra/920_join_headscale_mesh.yml --limit <target-host>
+    ```
+* The playbook will:
+    * Generate an ephemeral pre-auth key (expires in 1 minute) by SSHing from lapy to the headscale server
+    * Install Tailscale on the target machine
+    * Configure Tailscale to connect to your headscale server
+    * Enable magic DNS so devices can talk to each other by hostname
+
+#### Manual method (for mobile apps, desktop clients, etc.)
+
 * Install Tailscale on your devices (mobile apps, desktop clients, etc.).
+* Generate a pre-auth key by SSHing into your headscale server:
+    ```bash
+    ssh <headscale-server>
+    sudo headscale preauthkeys create --user counter-net --reusable
+    ```
 * Instead of using the default Tailscale login, use your headscale server:
     * Server URL: `https://headscale.contrapeso.xyz` (or your configured domain)
     * Use the pre-auth key you generated above
@@ -263,3 +294,31 @@ Headscale is a self-hosted Tailscale control server that allows you to create yo
 * View users: `headscale users list`
 * Generate new pre-auth keys: `headscale preauthkeys create --user counter-net --reusable`
 * Remove a device: `headscale nodes delete --identifier <node-id>`
+
+## Personal Blog
+
+Personal Blog is a static site served by Caddy's file server.
+
+### Deploy
+
+* Decide what subdomain you want to serve the personal blog on and add it to `ansible/services_config.yml` under `personal_blog`.
+    * Note that you will have to add a DNS entry to point to the VPS public IP.
+* Run the deployment playbook: `ansible-playbook -i inventory.ini services/personal-blog/deploy_personal_blog_playbook.yml`.
+* The playbook will:
+    * Create the web root directory at `/var/www/pablohere.contrapeso.xyz` (or your configured domain)
+    * Set up Caddy configuration with file_server directive
+    * Create an Uptime Kuma monitor for the site
+    * Configure proper permissions for deployment
+
+### Set up deployment alias on Lapy
+
+* Run the deployment alias setup playbook: `ansible-playbook -i inventory.ini services/personal-blog/setup_deploy_alias_lapy.yml`.
+* This creates a `deploy-personal-blog` alias in your `.bashrc` that allows you to deploy your static site from `~/pablohere/public/` to the server.
+* Source your `.bashrc` or open a new terminal to use the alias: `source ~/.bashrc`
+* Deploy your site by running: `deploy-personal-blog`
+    * The alias copies files via scp to a temporary location, then moves them with sudo to the web root and fixes permissions.
+
+Checklist:
+- [ ] Personal blog is accessible at the FQDN
+- [ ] Uptime Kuma monitor for the blog is showing as healthy
+- [ ] Deployment alias is working and you can successfully deploy files

SCRIPT_PLAYBOOK_MAPPING.md

@@ -0,0 +1,59 @@
+# Script to Playbook Mapping
+
+This document describes which playbooks each setup script applies to which machines.
+
+## Table
+
+| Script | Playbook | Target Machines/Groups | Notes |
+|--------|----------|------------------------|-------|
+| **setup_layer_0.sh** | None | N/A | Initial setup script - creates venv, config files |
+| **setup_layer_1a_vps.sh** | `infra/01_user_and_access_setup_playbook.yml` | `vps` (vipy, watchtower, spacey) | Creates counterweight user, configures SSH |
+| **setup_layer_1a_vps.sh** | `infra/02_firewall_and_fail2ban_playbook.yml` | `vps` (vipy, watchtower, spacey) | Configures UFW firewall and fail2ban |
+| **setup_layer_1b_nodito.sh** | `infra/nodito/30_proxmox_bootstrap_playbook.yml` | `nodito_host` (nodito) | Initial Proxmox bootstrap |
+| **setup_layer_1b_nodito.sh** | `infra/nodito/31_proxmox_community_repos_playbook.yml` | `nodito_host` (nodito) | Configures Proxmox community repositories |
+| **setup_layer_1b_nodito.sh** | `infra/nodito/32_zfs_pool_setup_playbook.yml` | `nodito_host` (nodito) | Sets up ZFS pool on Proxmox |
+| **setup_layer_1b_nodito.sh** | `infra/nodito/33_proxmox_debian_cloud_template.yml` | `nodito_host` (nodito) | Creates Debian cloud template for VMs |
+| **setup_layer_2.sh** | `infra/900_install_rsync.yml` | `all` (vipy, watchtower, spacey, nodito) | Installs rsync on all machines |
+| **setup_layer_2.sh** | `infra/910_docker_playbook.yml` | `all` (vipy, watchtower, spacey, nodito) | Installs Docker on all machines |
+| **setup_layer_3_caddy.sh** | `services/caddy_playbook.yml` | `vps` (vipy, watchtower, spacey) | Installs and configures Caddy reverse proxy |
+| **setup_layer_4_monitoring.sh** | `services/ntfy/deploy_ntfy_playbook.yml` | `watchtower` | Deploys ntfy notification service |
+| **setup_layer_4_monitoring.sh** | `services/uptime_kuma/deploy_uptime_kuma_playbook.yml` | `watchtower` | Deploys Uptime Kuma monitoring |
+| **setup_layer_4_monitoring.sh** | `services/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml` | `lapy` (localhost) | Configures backup of Uptime Kuma to laptop |
+| **setup_layer_4_monitoring.sh** | `services/ntfy/setup_ntfy_uptime_kuma_notification.yml` | `watchtower` | Configures ntfy notifications for Uptime Kuma |
+| **setup_layer_5_headscale.sh** | `services/headscale/deploy_headscale_playbook.yml` | `spacey` | Deploys Headscale mesh VPN server |
+| **setup_layer_5_headscale.sh** | `infra/920_join_headscale_mesh.yml` | `all` (vipy, watchtower, spacey, nodito) | Joins all machines to Headscale mesh (with --limit) |
+| **setup_layer_5_headscale.sh** | `services/headscale/setup_backup_headscale_to_lapy.yml` | `lapy` (localhost) | Configures backup of Headscale to laptop |
+| **setup_layer_6_infra_monitoring.sh** | `infra/410_disk_usage_alerts.yml` | `all` (vipy, watchtower, spacey, nodito, lapy) | Sets up disk usage monitoring alerts |
+| **setup_layer_6_infra_monitoring.sh** | `infra/420_system_healthcheck.yml` | `all` (vipy, watchtower, spacey, nodito, lapy) | Sets up system health checks |
+| **setup_layer_6_infra_monitoring.sh** | `infra/430_cpu_temp_alerts.yml` | `nodito_host` (nodito) | Sets up CPU temperature alerts for Proxmox |
+| **setup_layer_7_services.sh** | `services/vaultwarden/deploy_vaultwarden_playbook.yml` | `vipy` | Deploys Vaultwarden password manager |
+| **setup_layer_7_services.sh** | `services/forgejo/deploy_forgejo_playbook.yml` | `vipy` | Deploys Forgejo Git server |
+| **setup_layer_7_services.sh** | `services/lnbits/deploy_lnbits_playbook.yml` | `vipy` | Deploys LNbits Lightning wallet |
+| **setup_layer_7_services.sh** | `services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml` | `lapy` (localhost) | Configures backup of Vaultwarden to laptop |
+| **setup_layer_7_services.sh** | `services/lnbits/setup_backup_lnbits_to_lapy.yml` | `lapy` (localhost) | Configures backup of LNbits to laptop |
+| **setup_layer_8_secondary_services.sh** | `services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml` | `vipy` | Deploys emergency ntfy app |
+| **setup_layer_8_secondary_services.sh** | `services/memos/deploy_memos_playbook.yml` | `memos-box` (VM on nodito) | Deploys Memos note-taking service |
+
+## Machine Groups Reference
+
+- **vps**: vipy, watchtower, spacey (VPS servers)
+- **nodito_host**: nodito (Proxmox server)
+- **nodito_vms**: memos-box and other VMs created on nodito
+- **lapy**: localhost (your laptop)
+- **all**: All machines in inventory
+- **watchtower**: Single VPS for monitoring services
+- **vipy**: Single VPS for main services
+- **spacey**: Single VPS for Headscale
+- **memos-box**: VM on nodito for Memos service
+
+## Notes
+
+- Scripts use `--limit` flag to restrict playbooks that target `all` to specific hosts
+- Backup playbooks run on `lapy` (localhost) to configure backup jobs
+- Some playbooks are optional and may be skipped if hosts aren't configured
+- Layer 0 is a prerequisite for all other layers
+
+
+
+
+

ansible/backup.infra_vars.yml

@@ -0,0 +1,4 @@
+new_user: counterweight
+ssh_port: 22
+allow_ssh_from: "any"
+root_domain: contrapeso.xyz

ansible/example.inventory.ini

@@ -1,8 +1,14 @@
-[vipy]
-your.vps.ip.here ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
+[vps]
+vipy ansible_host=your.services.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
+watchtower ansible_host=your.monitoring.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
+spacey ansible_host=your.headscale.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
 
-[watchtower]
-your.vps.ip.here ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
+[nodito_host]
+nodito ansible_host=your.proxmox.ip.here ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key ansible_ssh_pass=your_root_password
+
+[nodito_vms]
+# Example node, replace with your VM names and addresses
+# memos_box ansible_host=192.168.1.150 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
 
 # Local connection to laptop: this assumes you're running ansible commands from your personal laptop
 # Make sure to adjust the username

ansible/infra/01_user_and_access_setup_playbook.yml

@@ -1,5 +1,5 @@
 - name: Secure Debian VPS
-  hosts: vipy,watchtower
+  hosts: vps
   vars_files:
     - ../infra_vars.yml
   become: true

ansible/infra/02_firewall_and_fail2ban_playbook.yml

@@ -1,5 +1,5 @@
 - name: Secure Debian VPS
-  hosts: vipy,watchtower
+  hosts: vps
   vars_files:
     - ../infra_vars.yml
   become: true

ansible/infra/410_disk_usage_alerts.yml

@@ -0,0 +1,330 @@
+- name: Deploy Disk Usage Monitoring
+  hosts: all
+  become: yes
+  vars_files:
+    - ../infra_vars.yml
+    - ../services_config.yml
+    - ../infra_secrets.yml
+
+  vars:
+    disk_usage_threshold_percent: 80
+    disk_check_interval_minutes: 15
+    monitored_mount_point: "/"
+    monitoring_script_dir: /opt/disk-monitoring
+    monitoring_script_path: "{{ monitoring_script_dir }}/disk_usage_monitor.sh"
+    log_file: "{{ monitoring_script_dir }}/disk_usage_monitor.log"
+    systemd_service_name: disk-usage-monitor
+    # Uptime Kuma configuration (auto-configured from services_config.yml and infra_secrets.yml)
+    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
+    ntfy_topic: "{{ service_settings.ntfy.topic }}"
+
+  tasks:
+    - name: Validate Uptime Kuma configuration
+      assert:
+        that:
+          - uptime_kuma_api_url is defined
+          - uptime_kuma_api_url != ""
+          - uptime_kuma_username is defined
+          - uptime_kuma_username != ""
+          - uptime_kuma_password is defined
+          - uptime_kuma_password != ""
+        fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
+
+    - name: Get hostname for monitor identification
+      command: hostname
+      register: host_name
+      changed_when: false
+
+    - name: Set monitor name and group based on hostname and mount point
+      set_fact:
+        monitor_name: "disk-usage-{{ host_name.stdout }}-{{ monitored_mount_point | replace('/', 'root') }}"
+        monitor_friendly_name: "Disk Usage: {{ host_name.stdout }} ({{ monitored_mount_point }})"
+        uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
+
+    - name: Create Uptime Kuma monitor setup script
+      copy:
+        dest: /tmp/setup_uptime_kuma_monitor.py
+        content: |
+          #!/usr/bin/env python3
+          import sys
+          import json
+          from uptime_kuma_api import UptimeKumaApi
+          
+          def main():
+              api_url = sys.argv[1]
+              username = sys.argv[2]
+              password = sys.argv[3]
+              group_name = sys.argv[4]
+              monitor_name = sys.argv[5]
+              monitor_description = sys.argv[6]
+              interval = int(sys.argv[7])
+              ntfy_topic = sys.argv[8] if len(sys.argv) > 8 else "alerts"
+              
+              api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
+              api.login(username, password)
+              
+              # Get all monitors
+              monitors = api.get_monitors()
+              
+              # Get all notifications and find ntfy notification
+              notifications = api.get_notifications()
+              ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
+              notification_id_list = {}
+              if ntfy_notification:
+                  notification_id_list[ntfy_notification['id']] = True
+              
+              # Find or create group
+              group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
+              if not group:
+                  group_result = api.add_monitor(type='group', name=group_name)
+                  # Refresh to get the full group object with id
+                  monitors = api.get_monitors()
+                  group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
+              
+              # Find or create/update push monitor
+              existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
+              
+              monitor_data = {
+                  'type': 'push',
+                  'name': monitor_name,
+                  'parent': group['id'],
+                  'interval': interval,
+                  'upsideDown': True,
+                  'description': monitor_description,
+                  'notificationIDList': notification_id_list
+              }
+              
+              if existing_monitor:
+                  monitor = api.edit_monitor(existing_monitor['id'], **monitor_data)
+                  # Refresh to get the full monitor object with pushToken
+                  monitors = api.get_monitors()
+                  monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
+              else:
+                  monitor_result = api.add_monitor(**monitor_data)
+                  # Refresh to get the full monitor object with pushToken
+                  monitors = api.get_monitors()
+                  monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
+              
+              # Output result as JSON
+              result = {
+                  'monitor_id': monitor['id'],
+                  'push_token': monitor['pushToken'],
+                  'group_name': group_name,
+                  'group_id': group['id'],
+                  'monitor_name': monitor_name
+              }
+              print(json.dumps(result))
+              
+              api.disconnect()
+          
+          if __name__ == '__main__':
+              main()
+        mode: '0755'
+      delegate_to: localhost
+      become: no
+
+    - name: Run Uptime Kuma monitor setup script
+      command: >
+        {{ ansible_playbook_python }}
+        /tmp/setup_uptime_kuma_monitor.py
+        "{{ uptime_kuma_api_url }}"
+        "{{ uptime_kuma_username }}"
+        "{{ uptime_kuma_password }}"
+        "{{ uptime_kuma_monitor_group }}"
+        "{{ monitor_name }}"
+        "{{ monitor_friendly_name }} - Alerts when usage exceeds {{ disk_usage_threshold_percent }}%"
+        "{{ (disk_check_interval_minutes * 60) + 60 }}"
+        "{{ ntfy_topic }}"
+      register: monitor_setup_result
+      delegate_to: localhost
+      become: no
+      changed_when: false
+
+    - name: Parse monitor setup result
+      set_fact:
+        monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
+
+    - name: Set push URL and monitor ID as facts
+      set_fact:
+        uptime_kuma_disk_usage_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
+        uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
+
+    - name: Install required packages for disk monitoring
+      package:
+        name:
+          - curl
+        state: present
+
+    - name: Create monitoring script directory
+      file:
+        path: "{{ monitoring_script_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Create disk usage monitoring script
+      copy:
+        dest: "{{ monitoring_script_path }}"
+        content: |
+          #!/bin/bash
+          
+          # Disk Usage Monitoring Script
+          # Monitors disk usage and sends alerts to Uptime Kuma
+          # Mode: "No news is good news" - only sends alerts when disk usage is HIGH
+          
+          LOG_FILE="{{ log_file }}"
+          USAGE_THRESHOLD="{{ disk_usage_threshold_percent }}"
+          UPTIME_KUMA_URL="{{ uptime_kuma_disk_usage_push_url }}"
+          MOUNT_POINT="{{ monitored_mount_point }}"
+          
+          # Function to log messages
+          log_message() {
+              echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
+          }
+          
+          # Function to get disk usage percentage
+          get_disk_usage() {
+              local mount_point="$1"
+              local usage=""
+              
+              # Get disk usage percentage (without % sign)
+              usage=$(df -h "$mount_point" 2>/dev/null | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
+              
+              if [ -z "$usage" ]; then
+                  log_message "ERROR: Could not read disk usage for $mount_point"
+                  return 1
+              fi
+              
+              echo "$usage"
+          }
+          
+          # Function to get disk usage details
+          get_disk_details() {
+              local mount_point="$1"
+              df -h "$mount_point" 2>/dev/null | awk 'NR==2 {print "Used: "$3" / Total: "$2" ("$5" full)"}'
+          }
+          
+          # Function to send alert to Uptime Kuma when disk usage exceeds threshold
+          # With upside-down mode enabled, sending status=up will trigger an alert
+          send_uptime_kuma_alert() {
+              local usage="$1"
+              local details="$2"
+              local message="DISK FULL WARNING: ${MOUNT_POINT} is ${usage}% full (Threshold: ${USAGE_THRESHOLD}%) - ${details}"
+              
+              log_message "ALERT: $message"
+              
+              # Send push notification to Uptime Kuma with status=up
+              # In upside-down mode, status=up is treated as down/alert
+              response=$(curl -s -w "\n%{http_code}" -G \
+                  --data-urlencode "status=up" \
+                  --data-urlencode "msg=$message" \
+                  "$UPTIME_KUMA_URL" 2>&1)
+              http_code=$(echo "$response" | tail -n1)
+              
+              if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
+                  log_message "Alert sent successfully to Uptime Kuma (HTTP $http_code)"
+              else
+                  log_message "ERROR: Failed to send alert to Uptime Kuma (HTTP $http_code)"
+              fi
+          }
+          
+          # Main monitoring logic
+          main() {
+              log_message "Starting disk usage check for $MOUNT_POINT"
+              
+              # Get current disk usage
+              current_usage=$(get_disk_usage "$MOUNT_POINT")
+              
+              if [ $? -ne 0 ] || [ -z "$current_usage" ]; then
+                  log_message "ERROR: Could not read disk usage"
+                  exit 1
+              fi
+              
+              # Get disk details
+              disk_details=$(get_disk_details "$MOUNT_POINT")
+              
+              log_message "Current disk usage: ${current_usage}% - $disk_details"
+              
+              # Check if usage exceeds threshold
+              if [ "$current_usage" -gt "$USAGE_THRESHOLD" ]; then
+                  log_message "WARNING: Disk usage ${current_usage}% exceeds threshold ${USAGE_THRESHOLD}%"
+                  send_uptime_kuma_alert "$current_usage" "$disk_details"
+              else
+                  log_message "Disk usage is within normal range - no alert needed (no news is good news)"
+              fi
+          }
+          
+          # Run main function
+          main
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Create systemd service for disk usage monitoring
+      copy:
+        dest: "/etc/systemd/system/{{ systemd_service_name }}.service"
+        content: |
+          [Unit]
+          Description=Disk Usage Monitor
+          After=network.target
+          
+          [Service]
+          Type=oneshot
+          ExecStart={{ monitoring_script_path }}
+          User=root
+          StandardOutput=journal
+          StandardError=journal
+          
+          [Install]
+          WantedBy=multi-user.target
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Create systemd timer for disk usage monitoring
+      copy:
+        dest: "/etc/systemd/system/{{ systemd_service_name }}.timer"
+        content: |
+          [Unit]
+          Description=Run Disk Usage Monitor every {{ disk_check_interval_minutes }} minute(s)
+          Requires={{ systemd_service_name }}.service
+          
+          [Timer]
+          OnBootSec={{ disk_check_interval_minutes }}min
+          OnUnitActiveSec={{ disk_check_interval_minutes }}min
+          Persistent=true
+          
+          [Install]
+          WantedBy=timers.target
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Reload systemd daemon
+      systemd:
+        daemon_reload: yes
+
+    - name: Enable and start disk usage monitoring timer
+      systemd:
+        name: "{{ systemd_service_name }}.timer"
+        enabled: yes
+        state: started
+
+    - name: Test disk usage monitoring script
+      command: "{{ monitoring_script_path }}"
+      register: script_test
+      changed_when: false
+
+    - name: Verify script execution
+      assert:
+        that:
+          - script_test.rc == 0
+        fail_msg: "Disk usage monitoring script failed to execute properly"
+
+    - name: Clean up temporary Uptime Kuma setup script
+      file:
+        path: /tmp/setup_uptime_kuma_monitor.py
+        state: absent
+      delegate_to: localhost
+      become: no

ansible/infra/420_system_healthcheck.yml

@@ -0,0 +1,312 @@
+- name: Deploy System Healthcheck Monitoring
+  hosts: all
+  become: yes
+  vars_files:
+    - ../infra_vars.yml
+    - ../services_config.yml
+    - ../infra_secrets.yml
+
+  vars:
+    healthcheck_interval_seconds: 60  # Send healthcheck every 60 seconds (1 minute)
+    healthcheck_timeout_seconds: 90   # Uptime Kuma should alert if no ping received within 90s
+    healthcheck_retries: 1            # Number of retries before alerting
+    monitoring_script_dir: /opt/system-healthcheck
+    monitoring_script_path: "{{ monitoring_script_dir }}/system_healthcheck.sh"
+    log_file: "{{ monitoring_script_dir }}/system_healthcheck.log"
+    systemd_service_name: system-healthcheck
+    # Uptime Kuma configuration (auto-configured from services_config.yml and infra_secrets.yml)
+    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
+    ntfy_topic: "{{ service_settings.ntfy.topic }}"
+
+  tasks:
+    - name: Validate Uptime Kuma configuration
+      assert:
+        that:
+          - uptime_kuma_api_url is defined
+          - uptime_kuma_api_url != ""
+          - uptime_kuma_username is defined
+          - uptime_kuma_username != ""
+          - uptime_kuma_password is defined
+          - uptime_kuma_password != ""
+        fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
+
+    - name: Get hostname for monitor identification
+      command: hostname
+      register: host_name
+      changed_when: false
+
+    - name: Set monitor name and group based on hostname
+      set_fact:
+        monitor_name: "system-healthcheck-{{ host_name.stdout }}"
+        monitor_friendly_name: "System Healthcheck: {{ host_name.stdout }}"
+        uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
+
+    - name: Create Uptime Kuma monitor setup script
+      copy:
+        dest: /tmp/setup_uptime_kuma_healthcheck_monitor.py
+        content: |
+          #!/usr/bin/env python3
+          import sys
+          import json
+          from uptime_kuma_api import UptimeKumaApi
+          
+          def main():
+              api_url = sys.argv[1]
+              username = sys.argv[2]
+              password = sys.argv[3]
+              group_name = sys.argv[4]
+              monitor_name = sys.argv[5]
+              monitor_description = sys.argv[6]
+              interval = int(sys.argv[7])
+              retries = int(sys.argv[8])
+              ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts"
+              
+              api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
+              api.login(username, password)
+              
+              # Get all monitors
+              monitors = api.get_monitors()
+              
+              # Get all notifications and find ntfy notification
+              notifications = api.get_notifications()
+              ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
+              notification_id_list = {}
+              if ntfy_notification:
+                  notification_id_list[ntfy_notification['id']] = True
+              
+              # Find or create group
+              group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
+              if not group:
+                  group_result = api.add_monitor(type='group', name=group_name)
+                  # Refresh to get the full group object with id
+                  monitors = api.get_monitors()
+                  group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
+              
+              # Find or create/update push monitor
+              existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
+              
+              monitor_data = {
+                  'type': 'push',
+                  'name': monitor_name,
+                  'parent': group['id'],
+                  'interval': interval,
+                  'upsideDown': False,  # Normal mode: receiving pings = healthy
+                  'maxretries': retries,
+                  'description': monitor_description,
+                  'notificationIDList': notification_id_list
+              }
+              
+              if existing_monitor:
+                  monitor = api.edit_monitor(existing_monitor['id'], **monitor_data)
+                  # Refresh to get the full monitor object with pushToken
+                  monitors = api.get_monitors()
+                  monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
+              else:
+                  monitor_result = api.add_monitor(**monitor_data)
+                  # Refresh to get the full monitor object with pushToken
+                  monitors = api.get_monitors()
+                  monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
+              
+              # Output result as JSON
+              result = {
+                  'monitor_id': monitor['id'],
+                  'push_token': monitor['pushToken'],
+                  'group_name': group_name,
+                  'group_id': group['id'],
+                  'monitor_name': monitor_name
+              }
+              print(json.dumps(result))
+              
+              api.disconnect()
+          
+          if __name__ == '__main__':
+              main()
+        mode: '0755'
+      delegate_to: localhost
+      become: no
+
+    - name: Run Uptime Kuma monitor setup script
+      command: >
+        {{ ansible_playbook_python }}
+        /tmp/setup_uptime_kuma_healthcheck_monitor.py
+        "{{ uptime_kuma_api_url }}"
+        "{{ uptime_kuma_username }}"
+        "{{ uptime_kuma_password }}"
+        "{{ uptime_kuma_monitor_group }}"
+        "{{ monitor_name }}"
+        "{{ monitor_friendly_name }} - Regular healthcheck ping every {{ healthcheck_interval_seconds }}s"
+        "{{ healthcheck_timeout_seconds }}"
+        "{{ healthcheck_retries }}"
+        "{{ ntfy_topic }}"
+      register: monitor_setup_result
+      delegate_to: localhost
+      become: no
+      changed_when: false
+
+    - name: Parse monitor setup result
+      set_fact:
+        monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
+
+    - name: Set push URL and monitor ID as facts
+      set_fact:
+        uptime_kuma_healthcheck_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
+        uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
+
+    - name: Install required packages for healthcheck monitoring
+      package:
+        name:
+          - curl
+        state: present
+
+    - name: Create monitoring script directory
+      file:
+        path: "{{ monitoring_script_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Create system healthcheck script
+      copy:
+        dest: "{{ monitoring_script_path }}"
+        content: |
+          #!/bin/bash
+          
+          # System Healthcheck Script
+          # Sends regular heartbeat pings to Uptime Kuma
+          # This ensures the system is running and able to communicate
+          
+          LOG_FILE="{{ log_file }}"
+          UPTIME_KUMA_URL="{{ uptime_kuma_healthcheck_push_url }}"
+          HOSTNAME=$(hostname)
+          
+          # Function to log messages
+          log_message() {
+              echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
+          }
+          
+          # Function to send healthcheck ping to Uptime Kuma
+          send_healthcheck() {
+              local uptime_seconds=$(awk '{print int($1)}' /proc/uptime)
+              local uptime_days=$((uptime_seconds / 86400))
+              local uptime_hours=$(((uptime_seconds % 86400) / 3600))
+              local uptime_minutes=$(((uptime_seconds % 3600) / 60))
+              
+              local message="System healthy - Uptime: ${uptime_days}d ${uptime_hours}h ${uptime_minutes}m"
+              
+              log_message "Sending healthcheck ping: $message"
+              
+              # Send push notification to Uptime Kuma with status=up
+              encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
+              response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
+              http_code=$(echo "$response" | tail -n1)
+              
+              if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
+                  log_message "Healthcheck ping sent successfully (HTTP $http_code)"
+              else
+                  log_message "ERROR: Failed to send healthcheck ping (HTTP $http_code)"
+                  return 1
+              fi
+          }
+          
+          # Main healthcheck logic
+          main() {
+              log_message "Starting system healthcheck for $HOSTNAME"
+              
+              # Send healthcheck ping
+              if send_healthcheck; then
+                  log_message "Healthcheck completed successfully"
+              else
+                  log_message "ERROR: Healthcheck failed"
+                  exit 1
+              fi
+          }
+          
+          # Run main function
+          main
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Create systemd service for system healthcheck
+      copy:
+        dest: "/etc/systemd/system/{{ systemd_service_name }}.service"
+        content: |
+          [Unit]
+          Description=System Healthcheck Monitor
+          After=network.target
+          
+          [Service]
+          Type=oneshot
+          ExecStart={{ monitoring_script_path }}
+          User=root
+          StandardOutput=journal
+          StandardError=journal
+          
+          [Install]
+          WantedBy=multi-user.target
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Create systemd timer for system healthcheck
+      copy:
+        dest: "/etc/systemd/system/{{ systemd_service_name }}.timer"
+        content: |
+          [Unit]
+          Description=Run System Healthcheck every minute
+          Requires={{ systemd_service_name }}.service
+          
+          [Timer]
+          OnBootSec=30sec
+          OnUnitActiveSec={{ healthcheck_interval_seconds }}sec
+          Persistent=true
+          
+          [Install]
+          WantedBy=timers.target
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Reload systemd daemon
+      systemd:
+        daemon_reload: yes
+
+    - name: Enable and start system healthcheck timer
+      systemd:
+        name: "{{ systemd_service_name }}.timer"
+        enabled: yes
+        state: started
+
+    - name: Test system healthcheck script
+      command: "{{ monitoring_script_path }}"
+      register: script_test
+      changed_when: false
+
+    - name: Verify script execution
+      assert:
+        that:
+          - script_test.rc == 0
+        fail_msg: "System healthcheck script failed to execute properly"
+
+    - name: Display monitor information
+      debug:
+        msg: |
+          ✓ System healthcheck monitoring deployed successfully!
+          
+          Monitor Name: {{ monitor_friendly_name }}
+          Monitor Group: {{ uptime_kuma_monitor_group }}
+          Healthcheck Interval: Every {{ healthcheck_interval_seconds }} seconds (1 minute)
+          Timeout: {{ healthcheck_timeout_seconds }} seconds (90s)
+          Retries: {{ healthcheck_retries }}
+          
+          The system will send a heartbeat ping every minute.
+          Uptime Kuma will alert if no ping is received within 90 seconds (with 1 retry).
+
+    - name: Clean up temporary Uptime Kuma setup script
+      file:
+        path: /tmp/setup_uptime_kuma_healthcheck_monitor.py
+        state: absent
+      delegate_to: localhost
+      become: no
+

ansible/infra/430_cpu_temp_alerts.yml

@@ -0,0 +1,316 @@
+- name: Deploy CPU Temperature Monitoring
+  hosts: nodito_host
+  become: yes
+  vars_files:
+    - ../infra_vars.yml
+    - ../services_config.yml
+    - ../infra_secrets.yml
+
+  vars:
+    temp_threshold_celsius: 80
+    temp_check_interval_minutes: 1
+    monitoring_script_dir: /opt/nodito-monitoring
+    monitoring_script_path: "{{ monitoring_script_dir }}/cpu_temp_monitor.sh"
+    log_file: "{{ monitoring_script_dir }}/cpu_temp_monitor.log"
+    systemd_service_name: nodito-cpu-temp-monitor
+    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
+    ntfy_topic: "{{ service_settings.ntfy.topic }}"
+
+  tasks:
+    - name: Validate Uptime Kuma configuration
+      assert:
+        that:
+          - uptime_kuma_api_url is defined
+          - uptime_kuma_api_url != ""
+          - uptime_kuma_username is defined
+          - uptime_kuma_username != ""
+          - uptime_kuma_password is defined
+          - uptime_kuma_password != ""
+        fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
+
+    - name: Get hostname for monitor identification
+      command: hostname
+      register: host_name
+      changed_when: false
+
+    - name: Set monitor name and group based on hostname
+      set_fact:
+        monitor_name: "cpu-temp-{{ host_name.stdout }}"
+        monitor_friendly_name: "CPU Temperature: {{ host_name.stdout }}"
+        uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
+
+    - name: Create Uptime Kuma CPU temperature monitor setup script
+      copy:
+        dest: /tmp/setup_uptime_kuma_cpu_temp_monitor.py
+        content: |
+          #!/usr/bin/env python3
+          import sys
+          import json
+          from uptime_kuma_api import UptimeKumaApi
+          
+          def main():
+              api_url = sys.argv[1]
+              username = sys.argv[2]
+              password = sys.argv[3]
+              group_name = sys.argv[4]
+              monitor_name = sys.argv[5]
+              monitor_description = sys.argv[6]
+              interval = int(sys.argv[7])
+              ntfy_topic = sys.argv[8] if len(sys.argv) > 8 else "alerts"
+          
+              api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
+              api.login(username, password)
+          
+              monitors = api.get_monitors()
+              notifications = api.get_notifications()
+          
+              ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
+              notification_id_list = {}
+              if ntfy_notification:
+                  notification_id_list[ntfy_notification['id']] = True
+          
+              group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
+              if not group:
+                  api.add_monitor(type='group', name=group_name)
+                  monitors = api.get_monitors()
+                  group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
+          
+              existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
+          
+              monitor_data = {
+                  'type': 'push',
+                  'name': monitor_name,
+                  'parent': group['id'],
+                  'interval': interval,
+                  'upsideDown': True,
+                  'description': monitor_description,
+                  'notificationIDList': notification_id_list
+              }
+          
+              if existing_monitor:
+                  api.edit_monitor(existing_monitor['id'], **monitor_data)
+              else:
+                  api.add_monitor(**monitor_data)
+          
+              monitors = api.get_monitors()
+              monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
+          
+              result = {
+                  'monitor_id': monitor['id'],
+                  'push_token': monitor['pushToken'],
+                  'group_name': group_name,
+                  'group_id': group['id'],
+                  'monitor_name': monitor_name
+              }
+              print(json.dumps(result))
+          
+              api.disconnect()
+          
+          if __name__ == '__main__':
+              main()
+        mode: '0755'
+      delegate_to: localhost
+      become: no
+
+    - name: Run Uptime Kuma monitor setup script
+      command: >
+        {{ ansible_playbook_python }}
+        /tmp/setup_uptime_kuma_cpu_temp_monitor.py
+        "{{ uptime_kuma_api_url }}"
+        "{{ uptime_kuma_username }}"
+        "{{ uptime_kuma_password }}"
+        "{{ uptime_kuma_monitor_group }}"
+        "{{ monitor_name }}"
+        "{{ monitor_friendly_name }} - Alerts when temperature exceeds {{ temp_threshold_celsius }}°C"
+        "{{ (temp_check_interval_minutes * 60) + 60 }}"
+        "{{ ntfy_topic }}"
+      register: monitor_setup_result
+      delegate_to: localhost
+      become: no
+      changed_when: false
+
+    - name: Parse monitor setup result
+      set_fact:
+        monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
+
+    - name: Set push URL and monitor ID as facts
+      set_fact:
+        uptime_kuma_cpu_temp_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
+        uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
+
+    - name: Install required packages for temperature monitoring
+      package:
+        name:
+          - lm-sensors
+          - curl
+          - jq
+          - bc
+        state: present
+
+    - name: Create monitoring script directory
+      file:
+        path: "{{ monitoring_script_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Create CPU temperature monitoring script
+      copy:
+        dest: "{{ monitoring_script_path }}"
+        content: |
+          #!/bin/bash
+          
+          # CPU Temperature Monitoring Script
+          # Monitors CPU temperature and sends alerts to Uptime Kuma
+          
+          LOG_FILE="{{ log_file }}"
+          TEMP_THRESHOLD="{{ temp_threshold_celsius }}"
+          UPTIME_KUMA_URL="{{ uptime_kuma_cpu_temp_push_url }}"
+          
+          log_message() {
+              echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
+          }
+          
+          get_cpu_temp() {
+              local temp=""
+          
+              if command -v sensors >/dev/null 2>&1; then
+                  temp=$(sensors 2>/dev/null | grep -E "Core 0|Package id 0|Tdie|Tctl" | head -1 | grep -oE '[0-9]+\.[0-9]+°C' | grep -oE '[0-9]+\.[0-9]+')
+              fi
+          
+              if [ -z "$temp" ] && [ -f /sys/class/thermal/thermal_zone0/temp ]; then
+                  temp=$(cat /sys/class/thermal/thermal_zone0/temp)
+                  temp=$(echo "scale=1; $temp/1000" | bc -l 2>/dev/null || echo "$temp")
+              fi
+          
+              if [ -z "$temp" ] && command -v acpi >/dev/null 2>&1; then
+                  temp=$(acpi -t 2>/dev/null | grep -oE '[0-9]+\.[0-9]+' | head -1)
+              fi
+          
+              echo "$temp"
+          }
+          
+          send_uptime_kuma_alert() {
+              local temp="$1"
+              local message="CPU Temperature Alert: ${temp}°C (Threshold: ${TEMP_THRESHOLD}°C)"
+          
+              log_message "ALERT: $message"
+          
+              encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/°/%C2%B0/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g')
+              response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
+              http_code=$(echo "$response" | tail -n1)
+          
+              if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
+                  log_message "Alert sent successfully to Uptime Kuma (HTTP $http_code)"
+              else
+                  log_message "ERROR: Failed to send alert to Uptime Kuma (HTTP $http_code)"
+              fi
+          }
+          
+          main() {
+              log_message "Starting CPU temperature check"
+          
+              current_temp=$(get_cpu_temp)
+          
+              if [ -z "$current_temp" ]; then
+                  log_message "ERROR: Could not read CPU temperature"
+                  exit 1
+              fi
+          
+              log_message "Current CPU temperature: ${current_temp}°C"
+          
+              if (( $(echo "$current_temp > $TEMP_THRESHOLD" | bc -l) )); then
+                  log_message "WARNING: CPU temperature ${current_temp}°C exceeds threshold ${TEMP_THRESHOLD}°C"
+                  send_uptime_kuma_alert "$current_temp"
+              else
+                  log_message "CPU temperature is within normal range"
+              fi
+          }
+          
+          main
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Create systemd service for CPU temperature monitoring
+      copy:
+        dest: "/etc/systemd/system/{{ systemd_service_name }}.service"
+        content: |
+          [Unit]
+          Description=CPU Temperature Monitor
+          After=network.target
+          
+          [Service]
+          Type=oneshot
+          ExecStart={{ monitoring_script_path }}
+          User=root
+          StandardOutput=journal
+          StandardError=journal
+          
+          [Install]
+          WantedBy=multi-user.target
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Create systemd timer for CPU temperature monitoring
+      copy:
+        dest: "/etc/systemd/system/{{ systemd_service_name }}.timer"
+        content: |
+          [Unit]
+          Description=Run CPU Temperature Monitor every {{ temp_check_interval_minutes }} minute(s)
+          Requires={{ systemd_service_name }}.service
+          
+          [Timer]
+          OnBootSec={{ temp_check_interval_minutes }}min
+          OnUnitActiveSec={{ temp_check_interval_minutes }}min
+          Persistent=true
+          
+          [Install]
+          WantedBy=timers.target
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Reload systemd daemon
+      systemd:
+        daemon_reload: yes
+
+    - name: Enable and start CPU temperature monitoring timer
+      systemd:
+        name: "{{ systemd_service_name }}.timer"
+        enabled: yes
+        state: started
+
+    - name: Test CPU temperature monitoring script
+      command: "{{ monitoring_script_path }}"
+      register: script_test
+      changed_when: false
+
+    - name: Verify script execution
+      assert:
+        that:
+          - script_test.rc == 0
+        fail_msg: "CPU temperature monitoring script failed to execute properly"
+
+    - name: Display monitoring configuration
+      debug:
+        msg:
+          - "CPU Temperature Monitoring configured successfully"
+          - "Temperature threshold: {{ temp_threshold_celsius }}°C"
+          - "Check interval: {{ temp_check_interval_minutes }} minute(s)"
+          - "Monitor Name: {{ monitor_friendly_name }}"
+          - "Monitor Group: {{ uptime_kuma_monitor_group }}"
+          - "Uptime Kuma Push URL: {{ uptime_kuma_cpu_temp_push_url }}"
+          - "Monitoring script: {{ monitoring_script_path }}"
+          - "Systemd Service: {{ systemd_service_name }}.service"
+          - "Systemd Timer: {{ systemd_service_name }}.timer"
+
+    - name: Clean up temporary Uptime Kuma setup script
+      file:
+        path: /tmp/setup_uptime_kuma_cpu_temp_monitor.py
+        state: absent
+      delegate_to: localhost
+      become: no
+

ansible/infra/920_join_headscale_mesh.yml

@@ -0,0 +1,118 @@
+- name: Join machine to headscale mesh network
+  hosts: all
+  become: yes
+  vars_files:
+    - ../infra_vars.yml
+    - ../services_config.yml
+  vars:
+    headscale_host_name: "spacey"
+    headscale_subdomain: "{{ subdomains.headscale }}"
+    headscale_domain: "https://{{ headscale_subdomain }}.{{ root_domain }}"
+    headscale_namespace: "{{ service_settings.headscale.namespace }}"
+
+  tasks:
+    - name: Set facts for headscale server connection
+      set_fact:
+        headscale_host: "{{ hostvars.get(headscale_host_name, {}).get('ansible_host', headscale_host_name) }}"
+        headscale_user: "{{ hostvars.get(headscale_host_name, {}).get('ansible_user', 'counterweight') }}"
+        headscale_key: "{{ hostvars.get(headscale_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+        headscale_port: "{{ hostvars.get(headscale_host_name, {}).get('ansible_port', 22) }}"
+
+    - name: Get user ID for namespace from headscale server via lapy
+      delegate_to: "{{ groups['lapy'][0] }}"
+      become: no
+      vars:
+        ssh_args: "{{ ('-i ' + headscale_key + ' ' if headscale_key else '') + '-p ' + headscale_port|string }}"
+      shell: >
+        ssh {{ ssh_args }}
+        {{ headscale_user }}@{{ headscale_host }}
+        "sudo headscale users list -o json"
+      register: users_list_result
+      changed_when: false
+      failed_when: users_list_result.rc != 0
+
+    - name: Extract user ID from users list
+      set_fact:
+        headscale_user_id: "{{ (users_list_result.stdout | from_json) | selectattr('name', 'equalto', headscale_namespace) | map(attribute='id') | first }}"
+      failed_when: headscale_user_id is not defined or headscale_user_id == ''
+
+    - name: Generate pre-auth key from headscale server via lapy
+      delegate_to: "{{ groups['lapy'][0] }}"
+      become: no
+      vars:
+        ssh_args: "{{ ('-i ' + headscale_key + ' ' if headscale_key else '') + '-p ' + headscale_port|string }}"
+      shell: >
+        ssh {{ ssh_args }}
+        {{ headscale_user }}@{{ headscale_host }}
+        "sudo headscale preauthkeys create --user {{ headscale_user_id }} --expiration 1m --output json"
+      register: preauth_key_result
+      changed_when: true
+      failed_when: preauth_key_result.rc != 0
+
+    - name: Extract auth key from preauth result
+      set_fact:
+        auth_key: "{{ (preauth_key_result.stdout | from_json).key }}"
+      failed_when: auth_key is not defined or auth_key == ''
+
+    - name: Install required packages for Tailscale
+      apt:
+        name:
+          - curl
+          - ca-certificates
+          - gnupg
+        state: present
+        update_cache: yes
+
+    - name: Create directory for GPG keyrings
+      file:
+        path: /etc/apt/keyrings
+        state: directory
+        mode: '0755'
+
+    - name: Download Tailscale GPG key
+      get_url:
+        url: https://pkgs.tailscale.com/stable/debian/bookworm.gpg
+        dest: /etc/apt/keyrings/tailscale.gpg
+        mode: '0644'
+
+    - name: Add Tailscale repository
+      apt_repository:
+        repo: "deb [signed-by=/etc/apt/keyrings/tailscale.gpg] https://pkgs.tailscale.com/stable/debian {{ ansible_lsb.codename }} main"
+        state: present
+        update_cache: yes
+
+    - name: Install Tailscale
+      apt:
+        name: tailscale
+        state: present
+        update_cache: yes
+
+    - name: Enable and start Tailscale service
+      systemd:
+        name: tailscaled
+        enabled: yes
+        state: started
+
+    - name: Configure Tailscale to use headscale server
+      command: >
+        tailscale up
+        --login-server {{ headscale_domain }}
+        --authkey {{ auth_key }}
+        --accept-dns=true
+      register: tailscale_up_result
+      changed_when: "'already authenticated' not in tailscale_up_result.stdout"
+      failed_when: tailscale_up_result.rc != 0 and 'already authenticated' not in tailscale_up_result.stdout
+
+    - name: Wait for Tailscale to be fully connected
+      pause:
+        seconds: 2
+
+    - name: Display Tailscale status
+      command: tailscale status
+      register: tailscale_status
+      changed_when: false
+
+    - name: Show Tailscale connection status
+      debug:
+        msg: "{{ tailscale_status.stdout_lines }}"
+

ansible/infra/nodito/30_proxmox_bootstrap_playbook.yml

@@ -0,0 +1,128 @@
+- name: Bootstrap Nodito SSH Key Access
+  hosts: nodito_host
+  become: true
+  vars_files:
+    - ../infra_vars.yml
+
+  tasks:
+    - name: Install sudo package
+      package:
+        name: sudo
+        state: present
+
+    - name: Ensure SSH directory exists for root
+      file:
+        path: /root/.ssh
+        state: directory
+        mode: "0700"
+        owner: root
+        group: root
+
+    - name: Install SSH public key for root
+      authorized_key:
+        user: root
+        key: "{{ lookup('file', ansible_ssh_private_key_file + '.pub') }}"
+        state: present
+
+    - name: Ensure SSH key-based authentication is enabled
+      lineinfile:
+        path: /etc/ssh/sshd_config
+        regexp: "^#?PubkeyAuthentication"
+        line: "PubkeyAuthentication yes"
+        state: present
+        backrefs: yes
+
+    - name: Ensure AuthorizedKeysFile is properly configured
+      lineinfile:
+        path: /etc/ssh/sshd_config
+        regexp: "^#?AuthorizedKeysFile"
+        line: "AuthorizedKeysFile .ssh/authorized_keys"
+        state: present
+        backrefs: yes
+
+    - name: Restart SSH service
+      service:
+        name: ssh
+        state: restarted
+
+    - name: Wait for SSH to be ready
+      wait_for:
+        port: "{{ ssh_port }}"
+        host: "{{ ansible_host }}"
+        delay: 2
+        timeout: 30
+
+    - name: Test SSH key authentication
+      command: whoami
+      register: ssh_key_test
+      changed_when: false
+
+    - name: Verify SSH key authentication works
+      assert:
+        that:
+          - ssh_key_test.stdout == "root"
+        fail_msg: "SSH key authentication failed - expected 'root', got '{{ ssh_key_test.stdout }}'"
+
+    - name: Create new user
+      user:
+        name: "{{ new_user }}"
+        groups: sudo
+        shell: /bin/bash
+        state: present
+        create_home: yes
+
+    - name: Set up SSH directory for new user
+      file:
+        path: "/home/{{ new_user }}/.ssh"
+        state: directory
+        mode: "0700"
+        owner: "{{ new_user }}"
+        group: "{{ new_user }}"
+
+    - name: Install SSH public key for new user
+      authorized_key:
+        user: "{{ new_user }}"
+        key: "{{ lookup('file', ansible_ssh_private_key_file + '.pub') }}"
+        state: present
+
+    - name: Allow new user to run sudo without password
+      copy:
+        dest: "/etc/sudoers.d/{{ new_user }}"
+        content: "{{ new_user }} ALL=(ALL) NOPASSWD:ALL"
+        owner: root
+        group: root
+        mode: "0440"
+
+    - name: Disable root login
+      lineinfile:
+        path: /etc/ssh/sshd_config
+        regexp: "^#?PermitRootLogin .*"
+        line: "PermitRootLogin no"
+        state: present
+        backrefs: yes
+
+    - name: Disable password authentication
+      lineinfile:
+        path: /etc/ssh/sshd_config
+        regexp: "^#?PasswordAuthentication .*"
+        line: "PasswordAuthentication no"
+        state: present
+        backrefs: yes
+
+    - name: Restart SSH service
+      service:
+        name: ssh
+        state: restarted
+
+    - name: Wait for SSH to be ready
+      wait_for:
+        port: "{{ ssh_port }}"
+        host: "{{ ansible_host }}"
+        delay: 2
+        timeout: 30
+
+    - name: Test connection with new user
+      command: whoami
+      become_user: "{{ new_user }}"
+      register: new_user_test
+      changed_when: false

ansible/infra/nodito/31_proxmox_community_repos_playbook.yml

@@ -0,0 +1,317 @@
+- name: Switch Proxmox VE from Enterprise to Community Repositories
+  hosts: nodito_host
+  become: true
+  vars_files:
+    - ../infra_vars.yml
+
+  tasks:
+    - name: Check for deb822 sources format
+      find:
+        paths: /etc/apt/sources.list.d/
+        patterns: "*.sources"
+        file_type: file
+      register: deb822_sources
+      changed_when: false
+
+    - name: Check for legacy .list files
+      find:
+        paths: /etc/apt/sources.list.d/
+        patterns: "*.list"
+        file_type: file
+      register: legacy_list_files
+      changed_when: false
+
+    - name: Check main sources.list for Proxmox entries
+      command: grep -q "proxmox\|trixie" /etc/apt/sources.list
+      register: main_sources_check
+      failed_when: false
+      changed_when: false
+
+    - name: Display current repository status
+      debug:
+        msg: |
+          Repository status:
+          - deb822 sources files: {{ deb822_sources.matched }}
+          - legacy .list files: {{ legacy_list_files.matched }}
+          - Proxmox/Trixie entries in sources.list: {{ main_sources_check.rc == 0 }}
+
+    - name: Check for enterprise repository in deb822 format
+      shell: |
+        for file in /etc/apt/sources.list.d/*.sources; do
+          if grep -q "Components:.*pve-enterprise" "$file" 2>/dev/null; then
+            echo "$file"
+            break
+          fi
+        done
+      register: enterprise_deb822_check
+      failed_when: false
+      changed_when: false
+
+    - name: Check for enterprise repository in legacy format
+      shell: |
+        for file in /etc/apt/sources.list.d/*.list; do
+          if grep -q "enterprise.proxmox.com" "$file" 2>/dev/null; then
+            echo "$file"
+            break
+          fi
+        done
+      register: enterprise_legacy_check
+      failed_when: false
+      changed_when: false
+
+    - name: Check for Ceph enterprise repository in deb822 format
+      shell: |
+        for file in /etc/apt/sources.list.d/*.sources; do
+          if grep -q "enterprise.proxmox.com.*ceph" "$file" 2>/dev/null; then
+            echo "$file"
+            break
+          fi
+        done
+      register: ceph_enterprise_deb822_check
+      failed_when: false
+      changed_when: false
+
+    - name: Check for Ceph enterprise repository in legacy format
+      shell: |
+        for file in /etc/apt/sources.list.d/*.list; do
+          if grep -q "enterprise.proxmox.com.*ceph" "$file" 2>/dev/null; then
+            echo "$file"
+            break
+          fi
+        done
+      register: ceph_enterprise_legacy_check
+      failed_when: false
+      changed_when: false
+
+    - name: Backup enterprise repository files
+      copy:
+        src: "{{ item }}"
+        dest: "{{ item }}.backup"
+        remote_src: yes
+        backup: yes
+      loop: "{{ (enterprise_deb822_check.stdout_lines + enterprise_legacy_check.stdout_lines + ceph_enterprise_deb822_check.stdout_lines + ceph_enterprise_legacy_check.stdout_lines) | select('string') | list }}"
+      when: (enterprise_deb822_check.stdout_lines + enterprise_legacy_check.stdout_lines + ceph_enterprise_deb822_check.stdout_lines + ceph_enterprise_legacy_check.stdout_lines) | select('string') | list | length > 0
+
+    - name: Delete enterprise repository files (deb822 format)
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop: "{{ enterprise_deb822_check.stdout_lines | select('string') | list }}"
+      when: enterprise_deb822_check.stdout_lines | select('string') | list | length > 0
+
+    - name: Delete enterprise repository files (legacy format)
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop: "{{ enterprise_legacy_check.stdout_lines | select('string') | list }}"
+      when: enterprise_legacy_check.stdout_lines | select('string') | list | length > 0
+
+    - name: Delete Ceph enterprise repository files (deb822 format)
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop: "{{ ceph_enterprise_deb822_check.stdout_lines | select('string') | list }}"
+      when: ceph_enterprise_deb822_check.stdout_lines | select('string') | list | length > 0
+
+    - name: Delete Ceph enterprise repository files (legacy format)
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop: "{{ ceph_enterprise_legacy_check.stdout_lines | select('string') | list }}"
+      when: ceph_enterprise_legacy_check.stdout_lines | select('string') | list | length > 0
+
+    - name: Create community repository file (deb822 format)
+      copy:
+        dest: /etc/apt/sources.list.d/proxmox.sources
+        content: |
+          Types: deb
+          URIs: http://download.proxmox.com/debian/pve
+          Suites: trixie
+          Components: pve-no-subscription
+          Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
+        owner: root
+        group: root
+        mode: '0644'
+        backup: yes
+      when: deb822_sources.matched > 0
+
+    - name: Create community repository file (legacy format)
+      copy:
+        dest: /etc/apt/sources.list.d/pve-no-subscription.list
+        content: |
+          # PVE pve-no-subscription repository provided by proxmox.com,
+          # NOT recommended for production use
+          deb http://download.proxmox.com/debian/pve trixie pve-no-subscription
+        owner: root
+        group: root
+        mode: '0644'
+        backup: yes
+      when: deb822_sources.matched == 0
+
+    - name: Create Ceph community repository file (deb822 format)
+      copy:
+        dest: /etc/apt/sources.list.d/ceph.sources
+        content: |
+          Types: deb
+          URIs: http://download.proxmox.com/debian/ceph-squid
+          Suites: trixie
+          Components: no-subscription
+          Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
+        owner: root
+        group: root
+        mode: '0644'
+        backup: yes
+      when: deb822_sources.matched > 0
+
+    - name: Create Ceph community repository file (legacy format)
+      copy:
+        dest: /etc/apt/sources.list.d/ceph-no-subscription.list
+        content: |
+          # Ceph no-subscription repository provided by proxmox.com,
+          # NOT recommended for production use
+          deb http://download.proxmox.com/debian/ceph-squid trixie no-subscription
+        owner: root
+        group: root
+        mode: '0644'
+        backup: yes
+      when: deb822_sources.matched == 0
+
+    - name: Update package cache
+      apt:
+        update_cache: yes
+        cache_valid_time: 3600
+
+    - name: Verify community repository is working
+      command: apt-cache policy proxmox-ve
+      register: community_repo_verify
+      changed_when: false
+
+    - name: Display community repository verification
+      debug:
+        var: community_repo_verify.stdout_lines
+
+    - name: Update Proxmox packages from community repository
+      apt:
+        name: proxmox-ve
+        state: latest
+        update_cache: yes
+
+    - name: Verify Proxmox VE version
+      command: pveversion
+      register: proxmox_version
+      changed_when: false
+
+    - name: Display Proxmox VE version
+      debug:
+        msg: "Proxmox VE version: {{ proxmox_version.stdout }}"
+
+    - name: Check repository status
+      shell: apt-cache policy | grep -A 5 -B 5 proxmox
+      register: final_repo_status
+      changed_when: false
+
+    - name: Display final repository status
+      debug:
+        var: final_repo_status.stdout_lines
+
+    - name: Verify no enterprise repository warnings
+      command: apt update
+      register: apt_update_result
+      changed_when: false
+
+    - name: Check for enterprise repository warnings
+      fail:
+        msg: "Enterprise repository warnings detected. Check the output above."
+      when: "'enterprise.proxmox.com' in apt_update_result.stdout"
+
+    - name: Create subscription nag removal script
+      copy:
+        dest: /usr/local/bin/pve-remove-nag.sh
+        content: |
+          #!/bin/sh
+          WEB_JS=/usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
+          if [ -s "$WEB_JS" ] && ! grep -q NoMoreNagging "$WEB_JS"; then
+              echo "Patching Web UI nag..."
+              sed -i -e "/data\.status/ s/!//" -e "/data\.status/ s/active/NoMoreNagging/" "$WEB_JS"
+          fi
+
+          MOBILE_TPL=/usr/share/pve-yew-mobile-gui/index.html.tpl
+          MARKER="<!-- MANAGED BLOCK FOR MOBILE NAG -->"
+          if [ -f "$MOBILE_TPL" ] && ! grep -q "$MARKER" "$MOBILE_TPL"; then
+              echo "Patching Mobile UI nag..."
+              printf "%s\n" \
+                "$MARKER" \
+                "<script>" \
+                "  function removeSubscriptionElements() {" \
+                "    // --- Remove subscription dialogs ---" \
+                "    const dialogs = document.querySelectorAll('dialog.pwt-outer-dialog');" \
+                "    dialogs.forEach(dialog => {" \
+                "      const text = (dialog.textContent || '').toLowerCase();" \
+                "      if (text.includes('subscription')) {" \
+                "        dialog.remove();" \
+                "        console.log('Removed subscription dialog');" \
+                "      }" \
+                "    });" \
+                "" \
+                "    // --- Remove subscription cards, but keep Reboot/Shutdown/Console ---" \
+                "    const cards = document.querySelectorAll('.pwt-card.pwt-p-2.pwt-d-flex.pwt-interactive.pwt-justify-content-center');" \
+                "    cards.forEach(card => {" \
+                "      const text = (card.textContent || '').toLowerCase();" \
+                "      const hasButton = card.querySelector('button');" \
+                "      if (!hasButton && text.includes('subscription')) {" \
+                "        card.remove();" \
+                "        console.log('Removed subscription card');" \
+                "      }" \
+                "    });" \
+                "  }" \
+                "" \
+                "  const observer = new MutationObserver(removeSubscriptionElements);" \
+                "  observer.observe(document.body, { childList: true, subtree: true });" \
+                "  removeSubscriptionElements();" \
+                "  setInterval(removeSubscriptionElements, 300);" \
+                "  setTimeout(() => {observer.disconnect();}, 10000);" \
+                "</script>" \
+                "" >> "$MOBILE_TPL"
+          fi
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Create APT configuration for nag removal
+      copy:
+        dest: /etc/apt/apt.conf.d/no-nag-script
+        content: |
+          DPkg::Post-Invoke { "/usr/local/bin/pve-remove-nag.sh"; };
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Run nag removal script immediately
+      command: /usr/local/bin/pve-remove-nag.sh
+      changed_when: false
+
+    - name: Reinstall proxmox-widget-toolkit to apply nag removal
+      apt:
+        name: proxmox-widget-toolkit
+        state: present
+        force: yes
+
+    - name: Clean up backup files
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /etc/apt/sources.list.d/ceph.sources.backup
+        - /etc/apt/sources.list.d/pve-enterprise.sources.backup
+      ignore_errors: yes
+
+    - name: Success message
+      debug:
+        msg: |
+          Successfully switched from Proxmox Enterprise to Community repositories.
+          Enterprise repository has been disabled and community repository is now active.
+          Subscription nag messages have been disabled.
+          Proxmox VE version: {{ proxmox_version.stdout }}
+          
+          IMPORTANT: Clear your browser cache or perform a hard reload (Ctrl+Shift+R) 
+          before using the Proxmox VE Web UI to avoid UI display issues.

ansible/infra/nodito/32_zfs_pool_setup_playbook.yml

@@ -0,0 +1,172 @@
+- name: Setup ZFS RAID 1 Pool for Proxmox Storage
+  hosts: nodito_host
+  become: true
+  vars_files:
+    - ../infra_vars.yml
+    - nodito_vars.yml
+
+  tasks:
+    - name: Verify Proxmox VE is running
+      command: pveversion
+      register: pve_version_check
+      changed_when: false
+      failed_when: pve_version_check.rc != 0
+
+    - name: Update package cache
+      apt:
+        update_cache: yes
+        cache_valid_time: 3600
+
+    - name: Install ZFS utilities
+      package:
+        name:
+          - zfsutils-linux
+          - zfs-initramfs
+        state: present
+
+    - name: Load ZFS kernel module
+      modprobe:
+        name: zfs
+
+    - name: Ensure ZFS module loads at boot
+      lineinfile:
+        path: /etc/modules
+        line: zfs
+        state: present
+
+    - name: Check if ZFS pool already exists
+      command: zpool list {{ zfs_pool_name }}
+      register: zfs_pool_exists
+      failed_when: false
+      changed_when: false
+
+    - name: Check if disks are in use
+      shell: |
+        for disk in {{ zfs_disk_1 }} {{ zfs_disk_2 }}; do
+          if mount | grep -q "^$disk"; then
+            echo "ERROR: $disk is mounted"
+            exit 1
+          fi
+          if lsblk -n -o MOUNTPOINT "$disk" | grep -v "^$" | grep -q .; then
+            echo "ERROR: $disk has mounted partitions"
+            exit 1
+          fi
+        done
+      register: disk_usage_check
+      failed_when: disk_usage_check.rc != 0
+      changed_when: false
+
+    - name: Create ZFS RAID 1 pool with optimized settings
+      command: >
+        zpool create {{ zfs_pool_name }}
+        -o ashift=12
+        -O mountpoint=none
+        mirror {{ zfs_disk_1 }} {{ zfs_disk_2 }}
+      when: zfs_pool_exists.rc != 0
+      register: zfs_pool_create_result
+
+    - name: Check if ZFS dataset already exists
+      command: zfs list {{ zfs_pool_name }}/vm-storage
+      register: zfs_dataset_exists
+      failed_when: false
+      changed_when: false
+
+    - name: Create ZFS dataset for Proxmox storage
+      command: zfs create {{ zfs_pool_name }}/vm-storage
+      when: zfs_dataset_exists.rc != 0
+      register: zfs_dataset_create_result
+
+    - name: Set ZFS dataset properties for Proxmox
+      command: zfs set {{ item.property }}={{ item.value }} {{ zfs_pool_name }}/vm-storage
+      loop:
+        - { property: "mountpoint", value: "{{ zfs_pool_mountpoint }}" }
+        - { property: "compression", value: "lz4" }
+        - { property: "atime", value: "off" }
+        - { property: "xattr", value: "sa" }
+        - { property: "acltype", value: "posixacl" }
+        - { property: "dnodesize", value: "auto" }
+      when: zfs_dataset_exists.rc != 0
+
+    - name: Set ZFS pool properties for Proxmox
+      command: zpool set autotrim=off {{ zfs_pool_name }}
+      when: zfs_pool_exists.rc != 0
+
+    - name: Set ZFS pool mountpoint for Proxmox
+      command: zfs set mountpoint={{ zfs_pool_mountpoint }} {{ zfs_pool_name }}
+      when: zfs_pool_exists.rc == 0
+
+    - name: Export and re-import ZFS pool for Proxmox compatibility
+      shell: |
+        zpool export {{ zfs_pool_name }}
+        zpool import {{ zfs_pool_name }}
+      when: zfs_pool_exists.rc != 0
+      register: zfs_pool_import_result
+
+
+    - name: Ensure ZFS services are enabled
+      systemd:
+        name: "{{ item }}"
+        enabled: yes
+        state: started
+      loop:
+        - zfs-import-cache
+        - zfs-import-scan
+        - zfs-mount
+        - zfs-share
+        - zfs-zed
+
+    - name: Check if ZFS pool storage already exists in Proxmox config
+      stat:
+        path: /etc/pve/storage.cfg
+      register: storage_cfg_file
+
+    - name: Check if storage name exists in Proxmox config
+      shell: "grep -q '^zfspool: {{ zfs_pool_name }}' /etc/pve/storage.cfg"
+      register: storage_exists_check
+      failed_when: false
+      changed_when: false
+      when: storage_cfg_file.stat.exists
+
+    - name: Set storage not configured when config file doesn't exist
+      set_fact:
+        storage_exists_check:
+          rc: 1
+      when: not storage_cfg_file.stat.exists
+
+    - name: Debug storage configuration status
+      debug:
+        msg: |
+          Config file exists: {{ storage_cfg_file.stat.exists }}
+          Storage check result: {{ storage_exists_check.rc }}
+          Pool exists: {{ zfs_pool_exists.rc == 0 }}
+          Will remove storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc == 0 }}
+          Will add storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc != 0 }}
+
+    - name: Remove existing storage if it exists
+      command: pvesm remove {{ zfs_pool_name }}
+      register: pvesm_remove_result
+      failed_when: false
+      when: 
+        - zfs_pool_exists.rc == 0
+        - storage_exists_check.rc == 0
+
+    - name: Add ZFS pool storage to Proxmox using pvesm
+      command: >
+        pvesm add zfspool {{ zfs_pool_name }}
+        --pool {{ zfs_pool_name }}
+        --content rootdir,images
+        --sparse 1
+      when: 
+        - zfs_pool_exists.rc == 0
+        - storage_exists_check.rc != 0
+      register: pvesm_add_result
+
+    - name: Verify ZFS pool is healthy
+      command: zpool status {{ zfs_pool_name }}
+      register: final_zfs_status
+      changed_when: false
+
+    - name: Fail if ZFS pool is not healthy
+      fail:
+        msg: "ZFS pool {{ zfs_pool_name }} is not in a healthy state"
+      when: "'ONLINE' not in final_zfs_status.stdout"

ansible/infra/nodito/33_proxmox_debian_cloud_template.yml

@@ -0,0 +1,186 @@
+- name: Create Proxmox template from Debian cloud image (no VM clone)
+  hosts: nodito_host
+  become: true
+  vars_files:
+    - ../../infra_vars.yml
+    - nodito_vars.yml
+
+  vars:
+    # Defaults (override via vars_files or --extra-vars as needed)
+    debian_cloud_image_url: "https://cloud.debian.org/images/cloud/trixie/20251006-2257/debian-13-genericcloud-amd64-20251006-2257.qcow2"
+    debian_cloud_image_filename: "debian-13-genericcloud-amd64-20251006-2257.qcow2"
+    debian_cloud_image_dest_dir: "/var/lib/vz/template/iso"
+    debian_cloud_image_dest_path: "{{ debian_cloud_image_dest_dir }}/{{ debian_cloud_image_filename }}"
+
+    proxmox_template_vmid: 9001
+    proxmox_template_name: "debian-13-cloud-init"
+    proxmox_template_memory_mb: 1024
+    proxmox_template_sockets: 1
+    proxmox_template_cores: 1
+    proxmox_template_bridge: "vmbr0"
+    proxmox_template_cpu_type: "host"
+    proxmox_template_disk_size_gb: 10
+
+    # Cloud-init defaults applied at template level (optional). You can override per-VM later.
+    proxmox_ciuser: "counterweight"            # Default login user to create; distro default may already exist
+    proxmox_sshkey_path: "/home/{{ new_user }}/.ssh/authorized_keys"  # Path to pubkey file for cloud-init injection
+    proxmox_ci_upgrade: true             # If true, run package upgrade on first boot
+
+    # Auto-install qemu-guest-agent in clones via cloud-init snippet
+    qemu_agent_snippet_filename: "user-data-qemu-agent.yaml"
+
+    # Storage to import disk into; use existing storage like local-lvm or your ZFS pool name
+    proxmox_image_storage: "{{ zfs_pool_name }}"
+
+  tasks:
+    - name: Verify Proxmox VE is running
+      command: pveversion
+      register: pve_version_check
+      changed_when: false
+      failed_when: pve_version_check.rc != 0
+
+    - name: Ensure destination directory exists for cloud image
+      file:
+        path: "{{ debian_cloud_image_dest_dir }}"
+        state: directory
+        mode: '0755'
+
+    - name: Check if Debian cloud image already present
+      stat:
+        path: "{{ debian_cloud_image_dest_path }}"
+      register: debian_image_stat
+
+    - name: Download Debian cloud image (qcow2)
+      get_url:
+        url: "{{ debian_cloud_image_url }}"
+        dest: "{{ debian_cloud_image_dest_path }}"
+        mode: '0644'
+        force: false
+      when: not debian_image_stat.stat.exists
+
+    - name: Ensure local storage allows snippets content (used for cloud-init snippets)
+      command: >
+        pvesm set local --content images,iso,vztmpl,snippets
+      failed_when: false
+
+    - name: Ensure snippets directory exists on storage mountpoint
+      file:
+        path: "{{ zfs_pool_mountpoint }}/snippets"
+        state: directory
+        mode: '0755'
+
+    - name: Read SSH public key content
+      slurp:
+        src: "{{ proxmox_sshkey_path }}"
+      register: ssh_key_content
+
+    - name: Extract SSH keys from authorized_keys file
+      set_fact:
+        ssh_keys_list: "{{ ssh_key_content.content | b64decode | split('\n') | select('match', '^ssh-') | list }}"
+
+    - name: Write cloud-init vendor-data snippet to install qemu-guest-agent
+      copy:
+        dest: "{{ zfs_pool_mountpoint }}/snippets/{{ qemu_agent_snippet_filename }}"
+        mode: '0644'
+        content: |
+          #cloud-config
+          # Vendor-data snippet: Proxmox will automatically set hostname from VM name when using vendor-data
+          # User info (ciuser/sshkeys) is set separately via Terraform/Proxmox parameters
+          package_update: true
+          package_upgrade: true
+          packages:
+            - qemu-guest-agent
+          runcmd:
+            - systemctl enable qemu-guest-agent
+            - systemctl start qemu-guest-agent
+
+    - name: Check if VMID already exists
+      command: qm config {{ proxmox_template_vmid }}
+      register: vmid_config_check
+      failed_when: false
+      changed_when: false
+
+    - name: Determine if VM is already a template
+      set_fact:
+        vm_already_template: "{{ 'template: 1' in vmid_config_check.stdout }}"
+      when: vmid_config_check.rc == 0
+
+    - name: Create base VM for template (no disk yet)
+      command: >
+        qm create {{ proxmox_template_vmid }}
+        --name {{ proxmox_template_name }}
+        --numa 0 --ostype l26
+        --cpu cputype={{ proxmox_template_cpu_type }}
+        --cores {{ proxmox_template_cores }}
+        --sockets {{ proxmox_template_sockets }}
+        --memory {{ proxmox_template_memory_mb }}
+        --net0 virtio,bridge={{ proxmox_template_bridge }}
+      when:
+        - vmid_config_check.rc != 0
+
+    - name: Import Debian cloud image as disk to storage
+      command: >
+        qm importdisk {{ proxmox_template_vmid }}
+        {{ debian_cloud_image_dest_path }}
+        {{ proxmox_image_storage }}
+      register: importdisk_result
+      changed_when: '"Successfully imported disk" in importdisk_result.stdout'
+      when:
+        - vmid_config_check.rc != 0 or not vm_already_template
+
+    - name: Check if ide2 (cloudinit) drive exists
+      command: qm config {{ proxmox_template_vmid }}
+      register: vm_config_check
+      failed_when: false
+      changed_when: false
+      when:
+        - vmid_config_check.rc == 0
+
+    - name: Remove existing ide2 (cloudinit) drive if it exists for idempotency
+      command: >
+        qm set {{ proxmox_template_vmid }} --delete ide2
+      register: ide2_removed
+      when:
+        - vmid_config_check.rc == 0
+        - "'ide2:' in vm_config_check.stdout"
+
+    - name: Build consolidated qm set argument list (simplified)
+      set_fact:
+        qm_set_args: >-
+          {{
+            [
+              '--scsihw virtio-scsi-pci',
+              '--scsi0 ' ~ proxmox_image_storage ~ ':vm-' ~ proxmox_template_vmid ~ '-disk-0',
+              '--ide2 ' ~ proxmox_image_storage ~ ':cloudinit',
+              '--ipconfig0 ip=dhcp',
+              '--boot c',
+              '--bootdisk scsi0',
+              '--serial0 socket',
+              '--vga serial0',
+              '--agent enabled=1',
+              '--ciuser ' ~ proxmox_ciuser,
+              '--sshkey ' ~ proxmox_sshkey_path
+            ]
+            + (proxmox_ci_upgrade | bool
+               | ternary(['--ciupgrade 1'], []))
+            + ['--cicustom vendor=local:snippets/' ~ qemu_agent_snippet_filename]
+          }}
+      when:
+        - vmid_config_check.rc != 0 or not vm_already_template | default(false) or ide2_removed.changed | default(false)
+
+    - name: Apply consolidated qm set
+      command: >
+        qm set {{ proxmox_template_vmid }} {{ qm_set_args | join(' ') }}
+      when:
+        - vmid_config_check.rc != 0 or not vm_already_template | default(false) or ide2_removed.changed | default(false)
+
+    - name: Resize primary disk to requested size
+      command: >
+        qm resize {{ proxmox_template_vmid }} scsi0 {{ proxmox_template_disk_size_gb }}G
+      when:
+        - vmid_config_check.rc != 0 or not vm_already_template
+
+    - name: Convert VM to template
+      command: qm template {{ proxmox_template_vmid }}
+      when:
+        - vmid_config_check.rc == 0 and not vm_already_template or vmid_config_check.rc != 0

ansible/infra/nodito/nodito_vars.yml

@@ -0,0 +1,19 @@
+# Nodito CPU Temperature Monitoring Configuration
+
+# Temperature Monitoring Configuration
+temp_threshold_celsius: 80
+temp_check_interval_minutes: 1
+
+# Script Configuration
+monitoring_script_dir: /opt/nodito-monitoring
+monitoring_script_path: "{{ monitoring_script_dir }}/cpu_temp_monitor.sh"
+log_file: "{{ monitoring_script_dir }}/cpu_temp_monitor.log"
+
+# System Configuration
+systemd_service_name: nodito-cpu-temp-monitor
+
+# ZFS Pool Configuration
+zfs_pool_name: "proxmox-tank-1"
+zfs_disk_1: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN0Z"  # First disk for RAID 1 mirror
+zfs_disk_2: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P"  # Second disk for RAID 1 mirror
+zfs_pool_mountpoint: "/var/lib/vz"

ansible/infra_secrets.yml.example

@@ -0,0 +1,11 @@
+# Uptime Kuma login credentials
+# Used by the disk monitoring playbook to create monitors automatically
+
+uptime_kuma_username: "admin"
+uptime_kuma_password: "your_password_here"
+
+# ntfy credentials
+# Used for notification channel setup in Uptime Kuma
+
+ntfy_username: "your_ntfy_username"
+ntfy_password: "your_ntfy_password"

ansible/infra_vars.yml

@@ -1,3 +1,6 @@
+# Infrastructure Variables
+# Generated by setup_layer_0.sh
+
 new_user: counterweight
 ssh_port: 22
 allow_ssh_from: "any"

ansible/requirements.yml

@@ -0,0 +1,11 @@
+---
+# Ansible Galaxy Collections Requirements
+# Install with: ansible-galaxy collection install -r requirements.yml
+
+collections:
+  # Uptime Kuma Ansible Collection
+  # Used by: infra/41_disk_usage_alerts.yml
+  # Provides modules to manage Uptime Kuma monitors programmatically
+  - name: lucasheld.uptime_kuma
+    version: ">=1.0.0"
+

ansible/services/caddy_playbook.yml

@@ -1,5 +1,5 @@
 - name: Install and configure Caddy on Debian 12
-  hosts: vipy,watchtower
+  hosts: vps
   become: yes
 
   tasks:

ansible/services/forgejo/deploy_forgejo_playbook.yml

@@ -3,9 +3,14 @@
   become: yes
   vars_files:
     - ../../infra_vars.yml
+    - ../../services_config.yml
+    - ../../infra_secrets.yml
     - ./forgejo_vars.yml
   vars:
+    forgejo_subdomain: "{{ subdomains.forgejo }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
     forgejo_domain: "{{ forgejo_subdomain }}.{{ root_domain }}"
+    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
 
   tasks:
     - name: Ensure required packages are installed
@@ -83,6 +88,22 @@
         enabled: yes
         state: started
 
+    - name: Ensure Caddy sites-enabled directory exists
+      file:
+        path: "{{ caddy_sites_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Ensure Caddyfile includes import directive for sites-enabled
+      lineinfile:
+        path: /etc/caddy/Caddyfile
+        line: 'import sites-enabled/*'
+        insertafter: EOF
+        state: present
+        backup: yes
+
     - name: Create Caddy reverse proxy configuration for forgejo
       copy:
         dest: "{{ caddy_sites_dir }}/forgejo.conf"
@@ -95,6 +116,110 @@
         mode: '0644'
 
     - name: Reload Caddy to apply new config
-      service:
-        name: caddy
-        state: reloaded
+      command: systemctl reload caddy
+
+    - name: Create Uptime Kuma monitor setup script for Forgejo
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/setup_forgejo_monitor.py
+        content: |
+          #!/usr/bin/env python3
+          import sys
+          import yaml
+          from uptime_kuma_api import UptimeKumaApi, MonitorType
+          
+          try:
+              with open('/tmp/ansible_config.yml', 'r') as f:
+                  config = yaml.safe_load(f)
+              
+              url = config['uptime_kuma_url']
+              username = config['username']
+              password = config['password']
+              monitor_url = config['monitor_url']
+              monitor_name = config['monitor_name']
+              
+              api = UptimeKumaApi(url, timeout=30)
+              api.login(username, password)
+              
+              # Get all monitors
+              monitors = api.get_monitors()
+              
+              # Find or create "services" group
+              group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              if not group:
+                  group_result = api.add_monitor(type='group', name='services')
+                  # Refresh to get the group with id
+                  monitors = api.get_monitors()
+                  group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              
+              # Check if monitor already exists
+              existing_monitor = None
+              for monitor in monitors:
+                  if monitor.get('name') == monitor_name:
+                      existing_monitor = monitor
+                      break
+              
+              # Get ntfy notification ID
+              notifications = api.get_notifications()
+              ntfy_notification_id = None
+              for notif in notifications:
+                  if notif.get('type') == 'ntfy':
+                      ntfy_notification_id = notif.get('id')
+                      break
+              
+              if existing_monitor:
+                  print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})")
+                  print("Skipping - monitor already configured")
+              else:
+                  print(f"Creating monitor '{monitor_name}'...")
+                  api.add_monitor(
+                      type=MonitorType.HTTP,
+                      name=monitor_name,
+                      url=monitor_url,
+                      parent=group['id'],
+                      interval=60,
+                      maxretries=3,
+                      retryInterval=60,
+                      notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {}
+                  )
+              
+              api.disconnect()
+              print("SUCCESS")
+              
+          except Exception as e:
+              print(f"ERROR: {str(e)}", file=sys.stderr)
+              sys.exit(1)
+        mode: '0755'
+
+    - name: Create temporary config for monitor setup
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/ansible_config.yml
+        content: |
+          uptime_kuma_url: "{{ uptime_kuma_api_url }}"
+          username: "{{ uptime_kuma_username }}"
+          password: "{{ uptime_kuma_password }}"
+          monitor_url: "https://{{ forgejo_domain }}/api/healthz"
+          monitor_name: "Forgejo"
+        mode: '0644'
+
+    - name: Run Uptime Kuma monitor setup
+      command: python3 /tmp/setup_forgejo_monitor.py
+      delegate_to: localhost
+      become: no
+      register: monitor_setup
+      changed_when: "'SUCCESS' in monitor_setup.stdout"
+      ignore_errors: yes
+
+    - name: Clean up temporary files
+      delegate_to: localhost
+      become: no
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /tmp/setup_forgejo_monitor.py
+        - /tmp/ansible_config.yml
+

ansible/services/forgejo/forgejo_vars.yml

@@ -9,11 +9,15 @@ forgejo_url: "https://codeberg.org/forgejo/forgejo/releases/download/v{{ forgejo
 forgejo_bin_path: "/usr/local/bin/forgejo"
 forgejo_user: "git"
 
-# Caddy
-caddy_sites_dir: /etc/caddy/sites-enabled
-forgejo_subdomain: forgejo
+# (caddy_sites_dir and subdomain now in services_config.yml)
 
 # Remote access
-remote_host: "{{ groups['vipy'][0] }}"
-remote_user: "{{ hostvars[remote_host]['ansible_user'] }}"
-remote_key_file: "{{ hostvars[remote_host]['ansible_ssh_private_key_file'] | default('') }}"
+remote_host_name: "vipy"
+remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
+remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
+remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
+
+# Local backup
+local_backup_dir: "{{ lookup('env', 'HOME') }}/forgejo-backups"
+backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/forgejo_backup.sh"

ansible/services/forgejo/setup_backup_forgejo_to_lapy.yml

@@ -0,0 +1,122 @@
+---
+- name: Configure local backup for Forgejo from remote
+  hosts: lapy
+  gather_facts: no
+  vars_files:
+    - ../../infra_vars.yml
+    - ./forgejo_vars.yml
+  vars:
+    remote_data_path: "{{ forgejo_data_dir }}"
+    remote_config_path: "{{ forgejo_config_dir }}"
+    forgejo_service_name: "forgejo"
+    gpg_recipient: "{{ hostvars['localhost']['gpg_recipient'] | default('') }}"
+    gpg_key_id: "{{ hostvars['localhost']['gpg_key_id'] | default('') }}"
+
+  tasks:
+    - name: Debug Forgejo backup vars
+      debug:
+        msg:
+          - "remote_host={{ remote_host }}"
+          - "remote_user={{ remote_user }}"
+          - "remote_data_path='{{ remote_data_path }}'"
+          - "remote_config_path='{{ remote_config_path }}'"
+          - "local_backup_dir={{ local_backup_dir }}"
+          - "gpg_recipient={{ gpg_recipient }}"
+          - "gpg_key_id={{ gpg_key_id }}"
+
+    - name: Ensure local backup directory exists
+      ansible.builtin.file:
+        path: "{{ local_backup_dir }}"
+        state: directory
+        mode: "0755"
+
+    - name: Ensure ~/.local/bin exists
+      ansible.builtin.file:
+        path: "{{ lookup('env', 'HOME') }}/.local/bin"
+        state: directory
+        mode: "0755"
+
+    - name: Create Forgejo backup script
+      ansible.builtin.copy:
+        dest: "{{ backup_script_path }}"
+        mode: "0750"
+        content: |
+          #!/bin/bash
+          set -euo pipefail
+
+          if [ -z "{{ gpg_recipient }}" ]; then
+            echo "GPG recipient is not configured. Aborting."
+            exit 1
+          fi
+
+          TIMESTAMP=$(date +'%Y-%m-%d')
+          ENCRYPTED_BACKUP="{{ local_backup_dir }}/forgejo-backup-$TIMESTAMP.tar.gz.gpg"
+
+          {% if remote_key_file %}
+          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}"
+          {% else %}
+          SSH_CMD="ssh -p {{ remote_port }}"
+          {% endif %}
+
+          echo "Stopping Forgejo service..."
+          $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl stop {{ forgejo_service_name }}"
+
+          echo "Creating encrypted backup archive..."
+          $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo tar -czf - {{ remote_data_path }} {{ remote_config_path }}" | \
+          gpg --batch --yes --encrypt --recipient "{{ gpg_recipient }}" --output "$ENCRYPTED_BACKUP"
+
+          echo "Starting Forgejo service..."
+          $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl start {{ forgejo_service_name }}"
+
+          # Rotate old backups (keep 3 days)
+          # Calculate cutoff date (3 days ago) and delete backups older than that
+          CUTOFF_DATE=$(date -d '3 days ago' +'%Y-%m-%d')
+          for backup_file in "{{ local_backup_dir }}"/forgejo-backup-*.tar.gz.gpg; do
+            if [ -f "$backup_file" ]; then
+              # Extract date from filename: forgejo-backup-YYYY-MM-DD.tar.gz.gpg
+              file_date=$(basename "$backup_file" | sed -n 's/forgejo-backup-\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\)\.tar\.gz\.gpg/\1/p')
+              if [ -n "$file_date" ] && [ "$file_date" != "$TIMESTAMP" ] && [ "$file_date" \< "$CUTOFF_DATE" ]; then
+                rm -f "$backup_file"
+              fi
+            fi
+          done
+
+          echo "Backup completed successfully"
+
+    - name: Ensure cronjob for Forgejo backup exists
+      ansible.builtin.cron:
+        name: "Forgejo backup"
+        user: "{{ lookup('env', 'USER') }}"
+        job: "{{ backup_script_path }}"
+        minute: 5
+        hour: "9,12,15,18"
+
+    - name: Run Forgejo backup script to create initial backup
+      ansible.builtin.command: "{{ backup_script_path }}"
+
+    - name: Verify backup was created
+      block:
+        - name: Get today's date
+          command: date +'%Y-%m-%d'
+          register: today_date
+          changed_when: false
+
+        - name: Check if backup file exists
+          stat:
+            path: "{{ local_backup_dir }}/forgejo-backup-{{ today_date.stdout }}.tar.gz.gpg"
+          register: backup_file_stat
+
+        - name: Verify backup file exists
+          assert:
+            that:
+              - backup_file_stat.stat.exists
+              - backup_file_stat.stat.isreg
+            fail_msg: "Backup file {{ local_backup_dir }}/forgejo-backup-{{ today_date.stdout }}.tar.gz.gpg was not created"
+            success_msg: "Backup file {{ local_backup_dir }}/forgejo-backup-{{ today_date.stdout }}.tar.gz.gpg exists"
+
+        - name: Verify backup file is not empty
+          assert:
+            that:
+              - backup_file_stat.stat.size > 0
+            fail_msg: "Backup file {{ local_backup_dir }}/forgejo-backup-{{ today_date.stdout }}.tar.gz.gpg exists but is empty"
+            success_msg: "Backup file size is {{ backup_file_stat.stat.size }} bytes"

ansible/services/headscale/deploy_headscale_playbook.yml

@@ -1,11 +1,18 @@
 - name: Deploy headscale and configure Caddy reverse proxy
-  hosts: vipy
+  hosts: spacey
   become: no
   vars_files:
     - ../../infra_vars.yml
+    - ../../services_config.yml
+    - ../../infra_secrets.yml
     - ./headscale_vars.yml
   vars:
+    headscale_subdomain: "{{ subdomains.headscale }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
     headscale_domain: "{{ headscale_subdomain }}.{{ root_domain }}"
+    headscale_base_domain: "tailnet.{{ root_domain }}"
+    headscale_namespace: "{{ service_settings.headscale.namespace }}"
+    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
 
   tasks:
     - name: Install required packages
@@ -34,6 +41,16 @@
         path: /tmp/headscale.deb
         state: absent
 
+    - name: Ensure headscale user exists
+      become: yes
+      user:
+        name: headscale
+        system: yes
+        shell: /usr/sbin/nologin
+        home: /var/lib/headscale
+        create_home: yes
+        state: present
+
     - name: Create headscale data directory
       become: yes
       file:
@@ -50,17 +67,7 @@
         state: directory
         owner: headscale
         group: headscale
-        mode: '0755'
-
-    - name: Ensure headscale user exists
-      become: yes
-      user:
-        name: headscale
-        system: yes
-        shell: /usr/sbin/nologin
-        home: /var/lib/headscale
-        create_home: yes
-        state: present
+        mode: '0770'
 
     - name: Ensure headscale user owns data directory
       become: yes
@@ -69,6 +76,14 @@
         owner: headscale
         group: headscale
         recurse: yes
+        mode: '0750'
+
+    - name: Add counterweight user to headscale group
+      become: yes
+      user:
+        name: counterweight
+        groups: headscale
+        append: yes
 
     - name: Create ACL policies file
       become: yes
@@ -84,7 +99,7 @@
           }
         owner: headscale
         group: headscale
-        mode: '0644'
+        mode: '0640'
       notify: Restart headscale
 
     - name: Deploy headscale configuration file
@@ -135,17 +150,17 @@
             path: /etc/headscale/acl.json
           
           dns:
-            base_domain: tailnet.contrapeso.xyz
+            base_domain: {{ headscale_base_domain | quote }}
             magic_dns: true
             search_domains:
-              - tailnet.contrapeso.xyz
+              - {{ headscale_base_domain | quote }}
             nameservers:
               global:
                 - 1.1.1.1
                 - 1.0.0.1
         owner: root
-        group: root
-        mode: '0644'
+        group: headscale
+        mode: '0640'
       notify: Restart headscale
 
     - name: Test headscale configuration
@@ -158,12 +173,47 @@
       debug:
         msg: "{{ headscale_config_test.stdout }}"
 
+    - name: Ensure headscale data directory has correct ownership before starting service
+      become: yes
+      file:
+        path: /var/lib/headscale
+        state: directory
+        owner: headscale
+        group: headscale
+        mode: '0750'
+        recurse: yes
+
+    - name: Ensure headscale run directory has correct ownership
+      become: yes
+      file:
+        path: /var/run/headscale
+        state: directory
+        owner: headscale
+        group: headscale
+        mode: '0770'
+
     - name: Enable and start headscale service
       become: yes
       systemd:
         name: headscale
         enabled: yes
         state: started
+        daemon_reload: yes
+
+    - name: Wait for headscale unix socket to be ready
+      become: yes
+      wait_for:
+        path: /var/run/headscale/headscale.sock
+        state: present
+        timeout: 60
+        delay: 2
+
+    - name: Create headscale namespace if it doesn't exist
+      become: yes
+      command: headscale users create {{ headscale_namespace }}
+      register: create_namespace_result
+      failed_when: create_namespace_result.rc != 0 and 'already exists' not in create_namespace_result.stderr and 'UNIQUE constraint' not in create_namespace_result.stderr
+      changed_when: create_namespace_result.rc == 0
 
     - name: Allow HTTPS through UFW
       become: yes
@@ -220,6 +270,111 @@
       become: yes
       command: systemctl reload caddy
 
+    - name: Create Uptime Kuma monitor setup script for Headscale
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/setup_headscale_monitor.py
+        content: |
+          #!/usr/bin/env python3
+          import sys
+          import yaml
+          from uptime_kuma_api import UptimeKumaApi, MonitorType
+          
+          try:
+              with open('/tmp/ansible_config.yml', 'r') as f:
+                  config = yaml.safe_load(f)
+              
+              url = config['uptime_kuma_url']
+              username = config['username']
+              password = config['password']
+              monitor_url = config['monitor_url']
+              monitor_name = config['monitor_name']
+              
+              api = UptimeKumaApi(url, timeout=30)
+              api.login(username, password)
+              
+              # Get all monitors
+              monitors = api.get_monitors()
+              
+              # Find or create "services" group
+              group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              if not group:
+                  group_result = api.add_monitor(type='group', name='services')
+                  # Refresh to get the group with id
+                  monitors = api.get_monitors()
+                  group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              
+              # Check if monitor already exists
+              existing_monitor = None
+              for monitor in monitors:
+                  if monitor.get('name') == monitor_name:
+                      existing_monitor = monitor
+                      break
+              
+              # Get ntfy notification ID
+              notifications = api.get_notifications()
+              ntfy_notification_id = None
+              for notif in notifications:
+                  if notif.get('type') == 'ntfy':
+                      ntfy_notification_id = notif.get('id')
+                      break
+              
+              if existing_monitor:
+                  print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})")
+                  print("Skipping - monitor already configured")
+              else:
+                  print(f"Creating monitor '{monitor_name}'...")
+                  api.add_monitor(
+                      type=MonitorType.HTTP,
+                      name=monitor_name,
+                      url=monitor_url,
+                      parent=group['id'],
+                      interval=60,
+                      maxretries=3,
+                      retryInterval=60,
+                      notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {}
+                  )
+              
+              api.disconnect()
+              print("SUCCESS")
+              
+          except Exception as e:
+              print(f"ERROR: {str(e)}", file=sys.stderr)
+              sys.exit(1)
+        mode: '0755'
+
+    - name: Create temporary config for monitor setup
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/ansible_config.yml
+        content: |
+          uptime_kuma_url: "{{ uptime_kuma_api_url }}"
+          username: "{{ uptime_kuma_username }}"
+          password: "{{ uptime_kuma_password }}"
+          monitor_url: "https://{{ headscale_domain }}/health"
+          monitor_name: "Headscale"
+        mode: '0644'
+
+    - name: Run Uptime Kuma monitor setup
+      command: python3 /tmp/setup_headscale_monitor.py
+      delegate_to: localhost
+      become: no
+      register: monitor_setup
+      changed_when: "'SUCCESS' in monitor_setup.stdout"
+      ignore_errors: yes
+
+    - name: Clean up temporary files
+      delegate_to: localhost
+      become: no
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /tmp/setup_headscale_monitor.py
+        - /tmp/ansible_config.yml
+
   handlers:
     - name: Restart headscale
       become: yes

ansible/services/headscale/headscale_vars.yml

@@ -1,21 +1,23 @@
 # Headscale service configuration
-headscale_subdomain: headscale
+# (subdomain and caddy_sites_dir now in services_config.yml)
+
 headscale_port: 8080
 headscale_grpc_port: 50443
 
 # Version
 headscale_version: "0.26.1"
 
-# Caddy
-caddy_sites_dir: /etc/caddy/sites-enabled
-
 # Data directory
 headscale_data_dir: /var/lib/headscale
 
+# Namespace now configured in services_config.yml under service_settings.headscale.namespace
+
 # Remote access
-remote_host: "{{ groups['vipy'][0] }}"
-remote_user: "{{ hostvars[remote_host]['ansible_user'] }}"
-remote_key_file: "{{ hostvars[remote_host]['ansible_ssh_private_key_file'] | default('') }}"
+remote_host_name: "spacey"
+remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
+remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
+remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
 
 # Local backup
 local_backup_dir: "{{ lookup('env', 'HOME') }}/headscale-backups"

ansible/services/headscale/setup_backup_headscale_to_lapy.yml

@@ -43,9 +43,9 @@
           mkdir -p "$BACKUP_DIR"
 
           {% if remote_key_file %}
-          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ hostvars[remote_host]['ansible_port'] | default(22) }}"
+          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}"
           {% else %}
-          SSH_CMD="ssh -p {{ hostvars[remote_host]['ansible_port'] | default(22) }}"
+          SSH_CMD="ssh -p {{ remote_port }}"
           {% endif %}
 
           # Stop headscale service for consistent backup

ansible/services/lnbits/deploy_lnbits_playbook.yml

@@ -3,9 +3,14 @@
   become: yes
   vars_files:
     - ../../infra_vars.yml
+    - ../../services_config.yml
+    - ../../infra_secrets.yml
     - ./lnbits_vars.yml
   vars:
+    lnbits_subdomain: "{{ subdomains.lnbits }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
     lnbits_domain: "{{ lnbits_subdomain }}.{{ root_domain }}"
+    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
 
   tasks:
     - name: Create lnbits directory
@@ -19,38 +24,41 @@
     - name: Install required system packages
       apt:
         name:
-          - python3.11
-          - python3.11-venv
+          - python3
           - python3-pip
+          - python3-venv
+          - python3-dev
           - git
           - curl
-          - build-essential
           - pkg-config
+          - build-essential
+          - libsecp256k1-dev
           - libffi-dev
+          - libgmp-dev
+          - libpq-dev
+          - automake
+          - autoconf
+          - libtool
+          - m4
+          - gawk
         state: present
         update_cache: yes
 
-    - name: Install Poetry
+    - name: Install uv packaging tool
       shell: |
-        curl -sSL https://install.python-poetry.org | python3 -
+        curl -LsSf https://astral.sh/uv/install.sh | sh
       args:
-        creates: "{{ lookup('env', 'HOME') }}/.local/bin/poetry"
-      become: yes
-      become_user: "{{ ansible_user }}"
-
-    - name: Add Poetry to PATH
-      lineinfile:
-        path: "{{ lookup('env', 'HOME') }}/.bashrc"
-        line: 'export PATH="$HOME/.local/bin:$PATH"'
-        state: present
+        creates: "/home/{{ ansible_user }}/.local/bin/uv"
       become: yes
       become_user: "{{ ansible_user }}"
+      environment:
+        HOME: "/home/{{ ansible_user }}"
 
     - name: Clone LNBits repository
       git:
         repo: https://github.com/lnbits/lnbits.git
         dest: "{{ lnbits_dir }}/lnbits"
-        version: main
+        version: "v1.3.1"
         accept_hostkey: yes
 
     - name: Change ownership of LNBits directory to user
@@ -60,10 +68,19 @@
         group: "{{ ansible_user }}"
         recurse: yes
 
-    - name: Install LNBits dependencies
-      command: $HOME/.local/bin/poetry install --only main
+    - name: Install LNBits dependencies with uv (Python 3.12)
+      command: /home/{{ ansible_user }}/.local/bin/uv sync --python 3.12 --all-extras --no-dev
       args:
         chdir: "{{ lnbits_dir }}/lnbits"
+      become: yes
+      become_user: "{{ ansible_user }}"
+      environment:
+        HOME: "/home/{{ ansible_user }}"
+        PATH: "/home/{{ ansible_user }}/.local/bin:/usr/local/bin:/usr/bin:/bin"
+        SECP_BUNDLED: "0"
+        PKG_CONFIG_PATH: "/usr/lib/x86_64-linux-gnu/pkgconfig"
+        ACLOCAL: "aclocal"
+        AUTOMAKE: "automake"
     
     - name: Copy .env.example to .env
       copy:
@@ -107,10 +124,12 @@
           Type=simple
           User={{ ansible_user }}
           WorkingDirectory={{ lnbits_dir }}/lnbits
-          ExecStart=/home/{{ ansible_user }}/.local/bin/poetry run lnbits
+          ExecStart=/home/{{ ansible_user }}/.local/bin/uv run --python 3.12 lnbits
           Restart=always
           RestartSec=30
           Environment=PYTHONUNBUFFERED=1
+          Environment="PATH=/home/{{ ansible_user }}/.local/bin:/usr/local/bin:/usr/bin:/bin"
+          Environment=SECP_BUNDLED=0
 
           [Install]
           WantedBy=multi-user.target
@@ -143,6 +162,8 @@
         insertafter: EOF
         state: present
         backup: yes
+        create: yes
+        mode: '0644'
 
     - name: Create Caddy reverse proxy configuration for lnbits
       copy:
@@ -159,3 +180,109 @@
 
     - name: Reload Caddy to apply new config
       command: systemctl reload caddy
+
+    - name: Create Uptime Kuma monitor setup script for LNBits
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/setup_lnbits_monitor.py
+        content: |
+          #!/usr/bin/env python3
+          import sys
+          import yaml
+          from uptime_kuma_api import UptimeKumaApi, MonitorType
+          
+          try:
+              with open('/tmp/ansible_config.yml', 'r') as f:
+                  config = yaml.safe_load(f)
+              
+              url = config['uptime_kuma_url']
+              username = config['username']
+              password = config['password']
+              monitor_url = config['monitor_url']
+              monitor_name = config['monitor_name']
+              
+              api = UptimeKumaApi(url, timeout=30)
+              api.login(username, password)
+              
+              # Get all monitors
+              monitors = api.get_monitors()
+              
+              # Find or create "services" group
+              group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              if not group:
+                  group_result = api.add_monitor(type='group', name='services')
+                  # Refresh to get the group with id
+                  monitors = api.get_monitors()
+                  group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              
+              # Check if monitor already exists
+              existing_monitor = None
+              for monitor in monitors:
+                  if monitor.get('name') == monitor_name:
+                      existing_monitor = monitor
+                      break
+              
+              # Get ntfy notification ID
+              notifications = api.get_notifications()
+              ntfy_notification_id = None
+              for notif in notifications:
+                  if notif.get('type') == 'ntfy':
+                      ntfy_notification_id = notif.get('id')
+                      break
+              
+              if existing_monitor:
+                  print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})")
+                  print("Skipping - monitor already configured")
+              else:
+                  print(f"Creating monitor '{monitor_name}'...")
+                  api.add_monitor(
+                      type=MonitorType.HTTP,
+                      name=monitor_name,
+                      url=monitor_url,
+                      parent=group['id'],
+                      interval=60,
+                      maxretries=3,
+                      retryInterval=60,
+                      notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {}
+                  )
+              
+              api.disconnect()
+              print("SUCCESS")
+              
+          except Exception as e:
+              print(f"ERROR: {str(e)}", file=sys.stderr)
+              sys.exit(1)
+        mode: '0755'
+
+    - name: Create temporary config for monitor setup
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/ansible_config.yml
+        content: |
+          uptime_kuma_url: "{{ uptime_kuma_api_url }}"
+          username: "{{ uptime_kuma_username }}"
+          password: "{{ uptime_kuma_password }}"
+          monitor_url: "https://{{ lnbits_domain }}/api/v1/health"
+          monitor_name: "LNBits"
+        mode: '0644'
+
+    - name: Run Uptime Kuma monitor setup
+      command: python3 /tmp/setup_lnbits_monitor.py
+      delegate_to: localhost
+      become: no
+      register: monitor_setup
+      changed_when: "'SUCCESS' in monitor_setup.stdout"
+      ignore_errors: yes
+
+    - name: Clean up temporary files
+      delegate_to: localhost
+      become: no
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /tmp/setup_lnbits_monitor.py
+        - /tmp/ansible_config.yml
+

ansible/services/lnbits/lnbits_vars.yml

@@ -3,14 +3,14 @@ lnbits_dir: /opt/lnbits
 lnbits_data_dir: "{{ lnbits_dir }}/data"
 lnbits_port: 8765
 
-# Caddy
-caddy_sites_dir: /etc/caddy/sites-enabled
-lnbits_subdomain: wallet
+# (caddy_sites_dir and subdomain now in services_config.yml)
 
 # Remote access
-remote_host: "{{ groups['vipy'][0] }}"
-remote_user: "{{ hostvars[remote_host]['ansible_user'] }}"
-remote_key_file: "{{ hostvars[remote_host]['ansible_ssh_private_key_file'] | default('') }}"
+remote_host_name: "vipy"
+remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
+remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
+remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
 
 # Local backup
 local_backup_dir: "{{ lookup('env', 'HOME') }}/lnbits-backups"

ansible/services/lnbits/setup_backup_lnbits_to_lapy.yml

@@ -45,9 +45,9 @@
           ENCRYPTED_BACKUP="{{ local_backup_dir }}/lnbits-backup-$TIMESTAMP.tar.gz.gpg"
 
           {% if remote_key_file %}
-          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ hostvars[remote_host]['ansible_port'] | default(22) }}"
+          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}"
           {% else %}
-          SSH_CMD="ssh -p {{ hostvars[remote_host]['ansible_port'] | default(22) }}"
+          SSH_CMD="ssh -p {{ remote_port }}"
           {% endif %}
 
           # Stop LNBits service before backup
@@ -68,9 +68,27 @@
           echo "Starting LNBits service..."
           $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl start lnbits.service"
 
-          # Rotate old encrypted backups (keep 14 days)
-          find "{{ local_backup_dir }}" -name "lnbits-backup-*.tar.gz.gpg" -mtime +13 -delete
-          find "{{ local_backup_dir }}" -name "lnbits-env-*.gpg" -mtime +13 -delete
+          # Rotate old backups (keep 14 days)
+          # Calculate cutoff date (14 days ago) and delete backups older than that
+          CUTOFF_DATE=$(date -d '14 days ago' +'%Y-%m-%d')
+          for backup_file in "{{ local_backup_dir }}"/lnbits-backup-*.tar.gz.gpg; do
+            if [ -f "$backup_file" ]; then
+              # Extract date from filename: lnbits-backup-YYYY-MM-DD.tar.gz.gpg
+              file_date=$(basename "$backup_file" | sed -n 's/lnbits-backup-\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\)\.tar\.gz\.gpg/\1/p')
+              if [ -n "$file_date" ] && [ "$file_date" != "$TIMESTAMP" ] && [ "$file_date" \< "$CUTOFF_DATE" ]; then
+                rm -f "$backup_file"
+              fi
+            fi
+          done
+          for env_file in "{{ local_backup_dir }}"/lnbits-env-*.gpg; do
+            if [ -f "$env_file" ]; then
+              # Extract date from filename: lnbits-env-YYYY-MM-DD.gpg
+              file_date=$(basename "$env_file" | sed -n 's/lnbits-env-\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\)\.gpg/\1/p')
+              if [ -n "$file_date" ] && [ "$file_date" != "$TIMESTAMP" ] && [ "$file_date" \< "$CUTOFF_DATE" ]; then
+                rm -f "$env_file"
+              fi
+            fi
+          done
 
           echo "Backup completed successfully"
 

ansible/services/memos/deploy_memos_playbook.yml

@@ -0,0 +1,175 @@
+- name: Deploy memos and configure Caddy reverse proxy
+  hosts: memos-box
+  become: yes
+  vars_files:
+    - ../../infra_vars.yml
+    - ../../services_config.yml
+    - ./memos_vars.yml
+  vars:
+    memos_subdomain: "{{ subdomains.memos }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
+    memos_domain: "{{ memos_subdomain }}.{{ root_domain }}"
+
+  tasks:
+    - name: Install required packages
+      apt:
+        name:
+          - wget
+          - curl
+          - unzip
+        state: present
+        update_cache: yes
+
+    - name: Get latest memos release version
+      uri:
+        url: https://api.github.com/repos/usememos/memos/releases/latest
+        return_content: yes
+      register: memos_latest_release
+
+    - name: Set memos version and find download URL
+      set_fact:
+        memos_version: "{{ memos_latest_release.json.tag_name | regex_replace('^v', '') }}"
+
+    - name: Find linux-amd64 download URL
+      set_fact:
+        memos_download_url: "{{ memos_latest_release.json.assets | json_query('[?contains(name, `linux-amd64`) && (contains(name, `.tar.gz`) || contains(name, `.zip`))].browser_download_url') | first }}"
+
+    - name: Display memos version to install
+      debug:
+        msg: "Installing memos version {{ memos_version }} from {{ memos_download_url }}"
+
+    - name: Download memos binary
+      get_url:
+        url: "{{ memos_download_url }}"
+        dest: /tmp/memos_archive
+        mode: '0644'
+      register: memos_download
+
+    - name: Extract memos binary
+      unarchive:
+        src: /tmp/memos_archive
+        dest: /tmp/memos_extract
+        remote_src: yes
+        creates: /tmp/memos_extract/memos
+
+    - name: Install memos binary
+      copy:
+        src: /tmp/memos_extract/memos
+        dest: /usr/local/bin/memos
+        mode: '0755'
+        remote_src: yes
+      notify: Restart memos
+
+    - name: Remove temporary files
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /tmp/memos_archive
+        - /tmp/memos_extract
+
+    - name: Ensure memos user exists
+      user:
+        name: memos
+        system: yes
+        shell: /usr/sbin/nologin
+        home: /var/lib/memos
+        create_home: yes
+        state: present
+
+    - name: Create memos data directory
+      file:
+        path: "{{ memos_data_dir }}"
+        state: directory
+        owner: memos
+        group: memos
+        mode: '0750'
+
+    - name: Create memos systemd service file
+      copy:
+        dest: /etc/systemd/system/memos.service
+        content: |
+          [Unit]
+          Description=memos service
+          After=network.target
+
+          [Service]
+          Type=simple
+          User=memos
+          Group=memos
+          ExecStart=/usr/local/bin/memos --port {{ memos_port }} --data {{ memos_data_dir }}
+          Restart=on-failure
+          RestartSec=5s
+
+          [Install]
+          WantedBy=multi-user.target
+        owner: root
+        group: root
+        mode: '0644'
+      notify: Restart memos
+
+    - name: Enable and start memos service
+      systemd:
+        name: memos
+        enabled: yes
+        state: started
+        daemon_reload: yes
+
+    - name: Wait for memos to be ready
+      uri:
+        url: "http://localhost:{{ memos_port }}/api/v1/status"
+        status_code: 200
+      register: memos_ready
+      until: memos_ready.status == 200
+      retries: 30
+      delay: 2
+      ignore_errors: yes
+
+    - name: Allow HTTPS through UFW
+      ufw:
+        rule: allow
+        port: '443'
+        proto: tcp
+
+    - name: Allow HTTP through UFW (for Let's Encrypt)
+      ufw:
+        rule: allow
+        port: '80'
+        proto: tcp
+
+    - name: Ensure Caddy sites-enabled directory exists
+      file:
+        path: "{{ caddy_sites_dir }}"
+        state: directory
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Ensure Caddyfile includes import directive for sites-enabled
+      lineinfile:
+        path: /etc/caddy/Caddyfile
+        line: 'import sites-enabled/*'
+        insertafter: EOF
+        state: present
+        backup: yes
+
+    - name: Create Caddy reverse proxy configuration for memos
+      copy:
+        dest: "{{ caddy_sites_dir }}/memos.conf"
+        content: |
+          {{ memos_domain }} {
+              reverse_proxy localhost:{{ memos_port }}
+          }
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Reload Caddy to apply new config
+      command: systemctl reload caddy
+
+  handlers:
+    - name: Restart memos
+      systemd:
+        name: memos
+        state: restarted
+

ansible/services/memos/memos_vars.yml

@@ -0,0 +1,18 @@
+# General
+memos_data_dir: /var/lib/memos
+memos_port: 5230
+
+# (caddy_sites_dir and subdomain now in services_config.yml)
+
+# Remote access
+remote_host_name: "memos-box"
+remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
+remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
+remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
+
+# Local backup
+local_backup_dir: "{{ lookup('env', 'HOME') }}/memos-backups"
+backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/memos_backup.sh"
+
+

ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml

@@ -3,9 +3,17 @@
   become: yes
   vars_files:
     - ../../infra_vars.yml
+    - ../../infra_secrets.yml
+    - ../../services_config.yml
     - ./ntfy_emergency_app_vars.yml
   vars:
+    ntfy_emergency_app_subdomain: "{{ subdomains.ntfy_emergency_app }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
     ntfy_emergency_app_domain: "{{ ntfy_emergency_app_subdomain }}.{{ root_domain }}"
+    ntfy_service_domain: "{{ subdomains.ntfy }}.{{ root_domain }}"
+    ntfy_emergency_app_ntfy_url: "https://{{ ntfy_service_domain }}"
+    ntfy_emergency_app_ntfy_user: "{{ ntfy_username | default('') }}"
+    ntfy_emergency_app_ntfy_password: "{{ ntfy_password | default('') }}"
 
   tasks:
     - name: Create ntfy-emergency-app directory

ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml

@@ -2,18 +2,15 @@
 ntfy_emergency_app_dir: /opt/ntfy-emergency-app
 ntfy_emergency_app_port: 3000
 
-# Caddy
-caddy_sites_dir: /etc/caddy/sites-enabled
-ntfy_emergency_app_subdomain: avisame
+# (caddy_sites_dir and subdomain now in services_config.yml)
 
 # ntfy configuration
 ntfy_emergency_app_topic: "emergencia"
-ntfy_emergency_app_ntfy_url: "https://ntfy.contrapeso.xyz"
-ntfy_emergency_app_ntfy_user: "counterweight"
-ntfy_emergency_app_ntfy_password: "superntfyme"
 ntfy_emergency_app_ui_message: "Leave Pablo a message, he will respond as soon as possible"
 
 # Remote access
-remote_host: "{{ groups['vipy'][0] }}"
-remote_user: "{{ hostvars[remote_host]['ansible_user'] }}"
-remote_key_file: "{{ hostvars[remote_host]['ansible_ssh_private_key_file'] | default('') }}"
+remote_host_name: "vipy"
+remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
+remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
+remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"

ansible/services/ntfy/deploy_ntfy_playbook.yml

@@ -3,8 +3,12 @@
   become: yes
   vars_files:
     - ../../infra_vars.yml
+    - ../../infra_secrets.yml
+    - ../../services_config.yml
     - ./ntfy_vars.yml
   vars:
+    ntfy_subdomain: "{{ subdomains.ntfy }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
     ntfy_domain: "{{ ntfy_subdomain }}.{{ root_domain }}"
 
   tasks:
@@ -70,7 +74,7 @@
     
     - name: Create ntfy admin user
       shell: |
-        (echo "{{ lookup('env', 'NTFY_PASSWORD') }}"; echo "{{ lookup('env', 'NTFY_PASSWORD') }}") | ntfy user add --role=admin "{{ lookup('env', 'NTFY_USER') }}"
+        (echo "{{ ntfy_password }}"; echo "{{ ntfy_password }}") | ntfy user add --role=admin "{{ ntfy_username }}"
 
     - name: Ensure Caddy sites-enabled directory exists
       file:

ansible/services/ntfy/ntfy_vars.yml

@@ -1,3 +1,3 @@
-caddy_sites_dir: /etc/caddy/sites-enabled
-ntfy_subdomain: ntfy
-ntfy_port: 6674
+ntfy_port: 6674
+
+# ntfy_topic now lives in services_config.yml under service_settings.ntfy.topic

ansible/services/ntfy/setup_ntfy_uptime_kuma_notification.yml

@@ -0,0 +1,155 @@
+- name: Setup ntfy as Uptime Kuma Notification Channel
+  hosts: watchtower
+  become: no
+  vars_files:
+    - ../../infra_vars.yml
+    - ../../services_config.yml
+    - ../../infra_secrets.yml
+    - ./ntfy_vars.yml
+
+  vars:
+    ntfy_subdomain: "{{ subdomains.ntfy }}"
+    ntfy_topic: "{{ service_settings.ntfy.topic }}"
+    uptime_kuma_subdomain: "{{ subdomains.uptime_kuma }}"
+    ntfy_domain: "{{ ntfy_subdomain }}.{{ root_domain }}"
+    ntfy_server_url: "https://{{ ntfy_domain }}"
+    ntfy_priority: 4  # 1=min, 2=low, 3=default, 4=high, 5=max
+    uptime_kuma_api_url: "https://{{ uptime_kuma_subdomain }}.{{ root_domain }}"
+
+  tasks:
+    - name: Validate Uptime Kuma configuration
+      assert:
+        that:
+          - uptime_kuma_api_url is defined
+          - uptime_kuma_api_url != ""
+          - uptime_kuma_username is defined
+          - uptime_kuma_username != ""
+          - uptime_kuma_password is defined
+          - uptime_kuma_password != ""
+        fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
+
+    - name: Validate ntfy configuration
+      assert:
+        that:
+          - ntfy_domain is defined
+          - ntfy_domain != ""
+          - ntfy_topic is defined
+          - ntfy_topic != ""
+          - ntfy_username is defined
+          - ntfy_username != ""
+          - ntfy_password is defined
+          - ntfy_password != ""
+        fail_msg: "ntfy_domain, ntfy_topic, ntfy_username and ntfy_password must be set"
+
+    - name: Create Uptime Kuma ntfy notification setup script
+      copy:
+        dest: /tmp/setup_uptime_kuma_ntfy_notification.py
+        content: |
+          #!/usr/bin/env python3
+          import sys
+          import json
+          from uptime_kuma_api import UptimeKumaApi
+          
+          def main():
+              api_url = sys.argv[1]
+              username = sys.argv[2]
+              password = sys.argv[3]
+              notification_name = sys.argv[4]
+              ntfy_server_url = sys.argv[5]
+              ntfy_topic = sys.argv[6]
+              ntfy_username = sys.argv[7]
+              ntfy_password = sys.argv[8]
+              ntfy_priority = int(sys.argv[9])
+              
+              api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
+              api.login(username, password)
+              
+              # Get all notifications
+              notifications = api.get_notifications()
+              
+              # Find existing ntfy notification by name
+              existing_notification = next((n for n in notifications if n.get('name') == notification_name), None)
+              
+              notification_data = {
+                  'name': notification_name,
+                  'type': 'ntfy',
+                  'isDefault': True,  # Apply to all monitors by default
+                  'applyExisting': True,  # Apply to existing monitors
+                  'ntfyserverurl': ntfy_server_url,
+                  'ntfytopic': ntfy_topic,
+                  'ntfyusername': ntfy_username,
+                  'ntfypassword': ntfy_password,
+                  'ntfyPriority': ntfy_priority
+              }
+              
+              if existing_notification:
+                  notification = api.edit_notification(existing_notification['id'], **notification_data)
+                  action = "updated"
+              else:
+                  notification = api.add_notification(**notification_data)
+                  action = "created"
+              
+              # Output result as JSON
+              result = {
+                  'notification_id': notification['id'],
+                  'notification_name': notification_name,
+                  'ntfy_server': ntfy_server_url,
+                  'ntfy_topic': ntfy_topic,
+                  'action': action
+              }
+              print(json.dumps(result))
+              
+              api.disconnect()
+          
+          if __name__ == '__main__':
+              main()
+        mode: '0755'
+      delegate_to: localhost
+      become: no
+
+    - name: Run Uptime Kuma ntfy notification setup script
+      command: >
+        {{ ansible_playbook_python }}
+        /tmp/setup_uptime_kuma_ntfy_notification.py
+        "{{ uptime_kuma_api_url }}"
+        "{{ uptime_kuma_username }}"
+        "{{ uptime_kuma_password }}"
+        "ntfy ({{ ntfy_topic }})"
+        "{{ ntfy_server_url }}"
+        "{{ ntfy_topic }}"
+        "{{ ntfy_username }}"
+        "{{ ntfy_password }}"
+        "{{ ntfy_priority }}"
+      register: notification_setup_result
+      delegate_to: localhost
+      become: no
+      changed_when: false
+
+    - name: Parse notification setup result
+      set_fact:
+        notification_info_parsed: "{{ notification_setup_result.stdout | from_json }}"
+
+    - name: Display notification information
+      debug:
+        msg: |
+          ✓ ntfy notification channel {{ notification_info_parsed.action }} successfully!
+          
+          Notification Name: ntfy ({{ ntfy_topic }})
+          ntfy Server: {{ ntfy_server_url }}
+          ntfy Topic: {{ ntfy_topic }}
+          Priority: {{ ntfy_priority }} (4=high)
+          Default for all monitors: Yes
+          Applied to existing monitors: Yes
+          
+          All Uptime Kuma monitors will now send alerts to your ntfy server
+          on the "{{ ntfy_topic }}" topic.
+          
+          You can subscribe to alerts at: {{ ntfy_server_url }}/{{ ntfy_topic }}
+
+    - name: Clean up temporary Uptime Kuma setup script
+      file:
+        path: /tmp/setup_uptime_kuma_ntfy_notification.py
+        state: absent
+      delegate_to: localhost
+      become: no
+

ansible/services/personal-blog/deploy_personal_blog_playbook.yml

@@ -1,53 +1,55 @@
-- name: Deploy personal blog static site
+- name: Deploy personal blog static site with Caddy file server
   hosts: vipy
   become: yes
   vars_files:
     - ../../infra_vars.yml
+    - ../../services_config.yml
+    - ../../infra_secrets.yml
     - ./personal_blog_vars.yml
+  vars:
+    personal_blog_subdomain: "{{ subdomains.personal_blog }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
+    personal_blog_domain: "{{ personal_blog_subdomain }}.{{ root_domain }}"
+    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
 
   tasks:
-    - name: Install git
-      apt:
-        name: git
-        state: present
+    - name: Ensure user is in www-data group
+      user:
+        name: "{{ ansible_user }}"
+        groups: www-data
+        append: yes
 
-    - name: Create source directory for blog
+    - name: Create web root directory for personal blog
       file:
-        path: "{{ personal_blog_source_dir }}"
+        path: "{{ personal_blog_web_root }}"
         state: directory
-        owner: root
-        group: root
-        mode: '0755'
-
-    - name: Create webroot directory
-      file:
-        path: "{{ personal_blog_webroot }}"
-        state: directory
-        owner: www-data
+        owner: "{{ ansible_user }}"
         group: www-data
-        mode: '0755'
+        mode: '2775'
 
-    - name: Clone blog repository with token authentication
-      git:
-        repo: "https://{{ personal_blog_git_username }}:{{ lookup('env', 'PERSONAL_BLOG_DEPLOY_TOKEN') }}@forgejo.contrapeso.xyz/counterweight/pablohere.git"
-        dest: "{{ personal_blog_source_dir }}"
-        version: master
-        force: yes
-      become_user: root
-
-    - name: Copy static files to webroot
+    - name: Fix ownership and permissions on web root directory
       shell: |
-        rsync -av --delete {{ personal_blog_source_dir }}/{{ personal_blog_source_folder }}/ {{ personal_blog_webroot }}/
-      args:
-        creates: "{{ personal_blog_webroot }}/index.html"
+        chown -R {{ ansible_user }}:www-data {{ personal_blog_web_root }}
+        find {{ personal_blog_web_root }} -type d -exec chmod 2775 {} \;
+        find {{ personal_blog_web_root }} -type f -exec chmod 664 {} \;
 
-    - name: Set ownership and permissions for webroot
-      file:
-        path: "{{ personal_blog_webroot }}"
-        owner: www-data
+    - name: Create placeholder index.html
+      copy:
+        dest: "{{ personal_blog_web_root }}/index.html"
+        content: |
+          <!DOCTYPE html>
+          <html>
+          <head>
+              <title>Personal Blog</title>
+          </head>
+          <body>
+              <h1>Personal Blog</h1>
+              <p>Site is ready. Deploy your static files here.</p>
+          </body>
+          </html>
+        owner: "{{ ansible_user }}"
         group: www-data
-        recurse: yes
-        state: directory
+        mode: '0664'
 
     - name: Ensure Caddy sites-enabled directory exists
       file:
@@ -65,12 +67,12 @@
         state: present
         backup: yes
 
-    - name: Create Caddy static site configuration
+    - name: Create Caddy file server configuration for personal blog
       copy:
         dest: "{{ caddy_sites_dir }}/personal-blog.conf"
         content: |
           {{ personal_blog_domain }} {
-              root * {{ personal_blog_webroot }}
+              root * {{ personal_blog_web_root }}
               file_server
           }
         owner: root
@@ -80,21 +82,108 @@
     - name: Reload Caddy to apply new config
       command: systemctl reload caddy
 
-    - name: Create update script for blog
+    - name: Create Uptime Kuma monitor setup script for Personal Blog
+      delegate_to: localhost
+      become: no
       copy:
-        dest: /usr/local/bin/update-personal-blog.sh
+        dest: /tmp/setup_personal_blog_monitor.py
         content: |
-          #!/bin/bash
-          cd {{ personal_blog_source_dir }}
-          git pull https://{{ personal_blog_git_username }}:${PERSONAL_BLOG_DEPLOY_TOKEN}@forgejo.contrapeso.xyz/counterweight/pablohere.git master
-          rsync -av --delete {{ personal_blog_source_dir }}/{{ personal_blog_source_folder }}/ {{ personal_blog_webroot }}/
-          chown -R www-data:www-data {{ personal_blog_webroot }}
-        owner: root
-        group: root
+          #!/usr/bin/env python3
+          import sys
+          import yaml
+          from uptime_kuma_api import UptimeKumaApi, MonitorType
+          
+          try:
+              with open('/tmp/ansible_config.yml', 'r') as f:
+                  config = yaml.safe_load(f)
+              
+              url = config['uptime_kuma_url']
+              username = config['username']
+              password = config['password']
+              monitor_url = config['monitor_url']
+              monitor_name = config['monitor_name']
+              
+              api = UptimeKumaApi(url, timeout=30)
+              api.login(username, password)
+              
+              # Get all monitors
+              monitors = api.get_monitors()
+              
+              # Find or create "services" group
+              group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              if not group:
+                  group_result = api.add_monitor(type='group', name='services')
+                  # Refresh to get the group with id
+                  monitors = api.get_monitors()
+                  group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              
+              # Check if monitor already exists
+              existing_monitor = None
+              for monitor in monitors:
+                  if monitor.get('name') == monitor_name:
+                      existing_monitor = monitor
+                      break
+              
+              # Get ntfy notification ID
+              notifications = api.get_notifications()
+              ntfy_notification_id = None
+              for notif in notifications:
+                  if notif.get('type') == 'ntfy':
+                      ntfy_notification_id = notif.get('id')
+                      break
+              
+              if existing_monitor:
+                  print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})")
+                  print("Skipping - monitor already configured")
+              else:
+                  print(f"Creating monitor '{monitor_name}'...")
+                  api.add_monitor(
+                      type=MonitorType.HTTP,
+                      name=monitor_name,
+                      url=monitor_url,
+                      parent=group['id'],
+                      interval=60,
+                      maxretries=3,
+                      retryInterval=60,
+                      notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {}
+                  )
+              
+              api.disconnect()
+              print("SUCCESS")
+              
+          except Exception as e:
+              print(f"ERROR: {str(e)}", file=sys.stderr)
+              sys.exit(1)
         mode: '0755'
 
-    - name: Add cron job to update blog every hour
-      cron:
-        name: "Update personal blog"
-        job: "0 * * * * PERSONAL_BLOG_DEPLOY_TOKEN={{ lookup('env', 'PERSONAL_BLOG_DEPLOY_TOKEN') }} /usr/local/bin/update-personal-blog.sh"
-        user: root
+    - name: Create temporary config for monitor setup
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/ansible_config.yml
+        content: |
+          uptime_kuma_url: "{{ uptime_kuma_api_url }}"
+          username: "{{ uptime_kuma_username }}"
+          password: "{{ uptime_kuma_password }}"
+          monitor_url: "https://{{ personal_blog_domain }}"
+          monitor_name: "Personal Blog"
+        mode: '0644'
+
+    - name: Run Uptime Kuma monitor setup
+      command: python3 /tmp/setup_personal_blog_monitor.py
+      delegate_to: localhost
+      become: no
+      register: monitor_setup
+      changed_when: "'SUCCESS' in monitor_setup.stdout"
+      ignore_errors: yes
+
+    - name: Clean up temporary files
+      delegate_to: localhost
+      become: no
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /tmp/setup_personal_blog_monitor.py
+        - /tmp/ansible_config.yml
+

ansible/services/personal-blog/personal_blog_vars.yml

@@ -1,8 +1,16 @@
-caddy_sites_dir: /etc/caddy/sites-enabled
-personal_blog_subdomain: pablohere
-personal_blog_domain: pablohere.contrapeso.xyz
-personal_blog_git_repo: https://forgejo.contrapeso.xyz/counterweight/pablohere.git
-personal_blog_git_username: counterweight
-personal_blog_source_dir: /opt/personal-blog
-personal_blog_webroot: /var/www/pablohere.contrapeso.xyz
-personal_blog_source_folder: public
+# Personal Blog Configuration
+
+# Web root directory on server
+personal_blog_web_root: "/var/www/pablohere.contrapeso.xyz"
+
+# Remote access for deployment
+remote_host_name: "vipy"
+remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
+remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
+remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
+
+# Local deployment paths
+local_source_dir: "{{ lookup('env', 'HOME') }}/pablohere/public"
+deploy_alias_name: "deploy-personal-blog"
+

ansible/services/personal-blog/setup_deploy_alias_lapy.yml

@@ -0,0 +1,33 @@
+- name: Configure deployment alias for personal blog in lapy .bashrc
+  hosts: lapy
+  gather_facts: no
+  vars_files:
+    - ../../infra_vars.yml
+    - ./personal_blog_vars.yml
+  vars:
+    bashrc_path: "{{ lookup('env', 'HOME') }}/.bashrc"
+    alias_line: "alias {{ deploy_alias_name }}='scp -r {{ local_source_dir }}/* {{ remote_user }}@{{ remote_host }}:/tmp/blog-deploy/ && ssh {{ remote_user }}@{{ remote_host }} \"sudo rm -rf {{ personal_blog_web_root }}/* && sudo cp -r /tmp/blog-deploy/* {{ personal_blog_web_root }}/ && sudo rm -rf /tmp/blog-deploy && sudo chown -R {{ remote_user }}:www-data {{ personal_blog_web_root }} && sudo find {{ personal_blog_web_root }} -type d -exec chmod 2775 {} \\; && sudo find {{ personal_blog_web_root }} -type f -exec chmod 664 {} \\;\"'"
+
+  tasks:
+    - name: Remove any existing deployment alias from .bashrc (to avoid duplicates)
+      lineinfile:
+        path: "{{ bashrc_path }}"
+        regexp: "^alias {{ deploy_alias_name }}="
+        state: absent
+        backup: yes
+
+    - name: Add or update deployment alias in .bashrc
+      lineinfile:
+        path: "{{ bashrc_path }}"
+        line: "{{ alias_line }}"
+        backup: yes
+        insertafter: EOF
+
+    - name: Display deployment alias information
+      debug:
+        msg:
+          - "Deployment alias '{{ deploy_alias_name }}' has been configured in {{ bashrc_path }}"
+          - "Usage: {{ deploy_alias_name }}"
+          - "This will scp {{ local_source_dir }}/* to {{ remote_user }}@{{ remote_host }}:{{ personal_blog_web_root }}/"
+          - "Note: You may need to run 'source ~/.bashrc' or open a new terminal to use the alias"
+

ansible/services/uptime_kuma/deploy_uptime_kuma_playbook.yml

@@ -3,8 +3,11 @@
   become: yes
   vars_files:
   - ../../infra_vars.yml
+  - ../../services_config.yml
   - ./uptime_kuma_vars.yml
   vars:
+    uptime_kuma_subdomain: "{{ subdomains.uptime_kuma }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
     uptime_kuma_domain: "{{ uptime_kuma_subdomain }}.{{ root_domain }}"
 
   tasks:

ansible/services/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml

@@ -43,15 +43,24 @@
           mkdir -p "$BACKUP_DIR"
 
           {% if remote_key_file %}
-          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ hostvars[remote_host]['ansible_port'] | default(22) }}"
+          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}"
           {% else %}
-          SSH_CMD="ssh -p {{ hostvars[remote_host]['ansible_port'] | default(22) }}"
+          SSH_CMD="ssh -p {{ remote_port }}"
           {% endif %}
 
           rsync -az -e "$SSH_CMD" --delete {{ remote_user }}@{{ remote_host }}:{{ remote_data_path }}/ "$BACKUP_DIR/"
 
           # Rotate old backups (keep 14 days)
-          find "{{ local_backup_dir }}" -maxdepth 1 -type d -name '20*' -mtime +13 -exec rm -rf {} \;
+          # Calculate cutoff date (14 days ago) and delete backups older than that
+          CUTOFF_DATE=$(date -d '14 days ago' +'%Y-%m-%d')
+          for dir in "{{ local_backup_dir }}"/20*; do
+            if [ -d "$dir" ]; then
+              dir_date=$(basename "$dir")
+              if [ "$dir_date" != "$TIMESTAMP" ] && [ "$dir_date" \< "$CUTOFF_DATE" ]; then
+                rm -rf "$dir"
+              fi
+            fi
+          done
 
     - name: Ensure cronjob for backup exists
       cron:
@@ -63,3 +72,36 @@
 
     - name: Run the backup script to make the first backup
       command: "{{ backup_script_path }}"
+
+    - name: Verify backup was created
+      block:
+        - name: Get today's date
+          command: date +'%Y-%m-%d'
+          register: today_date
+          changed_when: false
+
+        - name: Check backup directory exists and contains files
+          stat:
+            path: "{{ local_backup_dir }}/{{ today_date.stdout }}"
+          register: backup_dir_stat
+
+        - name: Verify backup directory exists
+          assert:
+            that:
+              - backup_dir_stat.stat.exists
+              - backup_dir_stat.stat.isdir
+            fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} was not created"
+            success_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists"
+
+        - name: Check if backup directory contains files
+          find:
+            paths: "{{ local_backup_dir }}/{{ today_date.stdout }}"
+            recurse: yes
+          register: backup_files
+
+        - name: Verify backup directory is not empty
+          assert:
+            that:
+              - backup_files.files | length > 0
+            fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists but is empty"
+            success_msg: "Backup directory contains {{ backup_files.files | length }} file(s)"

ansible/services/uptime_kuma/uptime_kuma_vars.yml

@@ -3,14 +3,12 @@ uptime_kuma_dir: /opt/uptime-kuma
 uptime_kuma_data_dir: "{{ uptime_kuma_dir }}/data"
 uptime_kuma_port: 3001
 
-# Caddy
-caddy_sites_dir: /etc/caddy/sites-enabled
-uptime_kuma_subdomain: uptime
-
 # Remote access
-remote_host: "{{ groups['watchtower'][0] }}"
-remote_user: "{{ hostvars[remote_host]['ansible_user'] }}"
-remote_key_file: "{{ hostvars[remote_host]['ansible_ssh_private_key_file'] | default('') }}"
+remote_host_name: "watchtower"
+remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
+remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
+remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
 
 # Local backup
 local_backup_dir: "{{ lookup('env', 'HOME') }}/uptime-kuma-backups"

ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml

@@ -3,9 +3,14 @@
   become: yes
   vars_files:
     - ../../infra_vars.yml
+    - ../../services_config.yml
+    - ../../infra_secrets.yml
     - ./vaultwarden_vars.yml
   vars:
+    vaultwarden_subdomain: "{{ subdomains.vaultwarden }}"
+    caddy_sites_dir: "{{ caddy_sites_dir }}"
     vaultwarden_domain: "{{ vaultwarden_subdomain }}.{{ root_domain }}"
+    uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
 
   tasks:
     - name: Create vaultwarden directory
@@ -106,3 +111,113 @@
     - name: Reload Caddy to apply new config
       command: systemctl reload caddy
 
+    - name: Create Uptime Kuma monitor setup script for Vaultwarden
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/setup_vaultwarden_monitor.py
+        content: |
+          #!/usr/bin/env python3
+          import sys
+          import traceback
+          import yaml
+          from uptime_kuma_api import UptimeKumaApi, MonitorType
+          
+          try:
+              # Load configs
+              with open('/tmp/ansible_config.yml', 'r') as f:
+                  config = yaml.safe_load(f)
+              
+              url = config['uptime_kuma_url']
+              username = config['username']
+              password = config['password']
+              monitor_url = config['monitor_url']
+              monitor_name = config['monitor_name']
+              
+              # Connect to Uptime Kuma
+              api = UptimeKumaApi(url, timeout=30)
+              api.login(username, password)
+              
+              # Get all monitors
+              monitors = api.get_monitors()
+              
+              # Find or create "services" group
+              group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              if not group:
+                  group_result = api.add_monitor(type='group', name='services')
+                  # Refresh to get the group with id
+                  monitors = api.get_monitors()
+                  group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
+              
+              # Check if monitor already exists
+              existing_monitor = None
+              for monitor in monitors:
+                  if monitor.get('name') == monitor_name:
+                      existing_monitor = monitor
+                      break
+              
+              # Get ntfy notification ID
+              notifications = api.get_notifications()
+              ntfy_notification_id = None
+              for notif in notifications:
+                  if notif.get('type') == 'ntfy':
+                      ntfy_notification_id = notif.get('id')
+                      break
+              
+              if existing_monitor:
+                  print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})")
+                  print("Skipping - monitor already configured")
+              else:
+                  print(f"Creating monitor '{monitor_name}'...")
+                  api.add_monitor(
+                      type=MonitorType.HTTP,
+                      name=monitor_name,
+                      url=monitor_url,
+                      parent=group['id'],
+                      interval=60,
+                      maxretries=3,
+                      retryInterval=60,
+                      notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {}
+                  )
+              
+              api.disconnect()
+              print("SUCCESS")
+              
+          except Exception as e:
+              error_msg = str(e) if str(e) else repr(e)
+              print(f"ERROR: {error_msg}", file=sys.stderr)
+              traceback.print_exc(file=sys.stderr)
+              sys.exit(1)
+        mode: '0755'
+
+    - name: Create temporary config for monitor setup
+      delegate_to: localhost
+      become: no
+      copy:
+        dest: /tmp/ansible_config.yml
+        content: |
+          uptime_kuma_url: "{{ uptime_kuma_api_url }}"
+          username: "{{ uptime_kuma_username }}"
+          password: "{{ uptime_kuma_password }}"
+          monitor_url: "https://{{ vaultwarden_domain }}/alive"
+          monitor_name: "Vaultwarden"
+        mode: '0644'
+
+    - name: Run Uptime Kuma monitor setup
+      command: python3 /tmp/setup_vaultwarden_monitor.py
+      delegate_to: localhost
+      become: no
+      register: monitor_setup
+      changed_when: "'SUCCESS' in monitor_setup.stdout"
+      ignore_errors: yes
+
+    - name: Clean up temporary files
+      delegate_to: localhost
+      become: no
+      file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /tmp/setup_vaultwarden_monitor.py
+        - /tmp/ansible_config.yml
+

ansible/services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml

@@ -41,15 +41,24 @@
           mkdir -p "$BACKUP_DIR"
 
           {% if remote_key_file %}
-          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ hostvars[remote_host]['ansible_port'] | default(22) }}"
+          SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}"
           {% else %}
-          SSH_CMD="ssh -p {{ hostvars[remote_host]['ansible_port'] | default(22) }}"
+          SSH_CMD="ssh -p {{ remote_port }}"
           {% endif %}
 
           rsync -az -e "$SSH_CMD" --delete {{ remote_user }}@{{ remote_host }}:{{ remote_data_path }}/ "$BACKUP_DIR/"
 
           # Rotate old backups (keep 14 days)
-          find "{{ local_backup_dir }}" -maxdepth 1 -type d -name '20*' -mtime +13 -exec rm -rf {} \;
+          # Calculate cutoff date (14 days ago) and delete backups older than that
+          CUTOFF_DATE=$(date -d '14 days ago' +'%Y-%m-%d')
+          for dir in "{{ local_backup_dir }}"/20*; do
+            if [ -d "$dir" ]; then
+              dir_date=$(basename "$dir")
+              if [ "$dir_date" != "$TIMESTAMP" ] && [ "$dir_date" \< "$CUTOFF_DATE" ]; then
+                rm -rf "$dir"
+              fi
+            fi
+          done
 
     - name: Ensure cronjob for backup exists
       cron:
@@ -61,3 +70,36 @@
 
     - name: Run the backup script to make the first backup
       command: "{{ backup_script_path }}"
+
+    - name: Verify backup was created
+      block:
+        - name: Get today's date
+          command: date +'%Y-%m-%d'
+          register: today_date
+          changed_when: false
+
+        - name: Check backup directory exists and contains files
+          stat:
+            path: "{{ local_backup_dir }}/{{ today_date.stdout }}"
+          register: backup_dir_stat
+
+        - name: Verify backup directory exists
+          assert:
+            that:
+              - backup_dir_stat.stat.exists
+              - backup_dir_stat.stat.isdir
+            fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} was not created"
+            success_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists"
+
+        - name: Check if backup directory contains files
+          find:
+            paths: "{{ local_backup_dir }}/{{ today_date.stdout }}"
+            recurse: yes
+          register: backup_files
+
+        - name: Verify backup directory is not empty
+          assert:
+            that:
+              - backup_files.files | length > 0
+            fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists but is empty"
+            success_msg: "Backup directory contains {{ backup_files.files | length }} file(s)"

ansible/services/vaultwarden/vaultwarden_vars.yml

@@ -3,14 +3,14 @@ vaultwarden_dir: /opt/vaultwarden
 vaultwarden_data_dir: "{{ vaultwarden_dir }}/data"
 vaultwarden_port: 8222
 
-# Caddy
-caddy_sites_dir: /etc/caddy/sites-enabled
-vaultwarden_subdomain: vault
+# (caddy_sites_dir and subdomain now in services_config.yml)
 
 # Remote access
-remote_host: "{{ groups['vipy'][0] }}"
-remote_user: "{{ hostvars[remote_host]['ansible_user'] }}"
-remote_key_file: "{{ hostvars[remote_host]['ansible_ssh_private_key_file'] | default('') }}"
+remote_host_name: "vipy"
+remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
+remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
+remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
+remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
 
 # Local backup
 local_backup_dir: "{{ lookup('env', 'HOME') }}/vaultwarden-backups"

ansible/services_config.yml

@@ -0,0 +1,33 @@
+# Centralized Services Configuration
+# Subdomains and Caddy settings for all services
+
+# Edit these subdomains to match your preferences
+subdomains:
+  # Monitoring Services (on watchtower)
+  ntfy: ntfy
+  uptime_kuma: uptime
+  
+  # VPN Infrastructure (on spacey)
+  headscale: headscale
+  
+  # Core Services (on vipy)
+  vaultwarden: vault
+  forgejo: forgejo
+  lnbits: wallet
+  
+  # Secondary Services (on vipy)
+  ntfy_emergency_app: emergency
+  personal_blog: pablohere
+  
+  # Memos (on memos-box)
+  memos: memos
+
+# Caddy configuration
+caddy_sites_dir: /etc/caddy/sites-enabled
+
+# Service-specific settings shared across playbooks
+service_settings:
+  ntfy:
+    topic: alerts
+  headscale:
+    namespace: counter-net

ansible/services_config.yml.example

@@ -0,0 +1,32 @@
+# Centralized Services Configuration
+# Copy this to services_config.yml and customize
+
+# Edit these subdomains to match your preferences
+subdomains:
+  # Monitoring Services (on watchtower)
+  ntfy: ntfy
+  uptime_kuma: uptime
+  
+  # VPN Infrastructure (on spacey)
+  headscale: headscale
+  
+  # Core Services (on vipy)
+  vaultwarden: vault
+  forgejo: git
+  lnbits: lnbits
+  
+  # Secondary Services (on vipy)
+  ntfy_emergency_app: emergency
+  
+  # Memos (on memos-box)
+  memos: memos
+
+# Caddy configuration
+caddy_sites_dir: /etc/caddy/sites-enabled
+
+# Service-specific settings shared across playbooks
+service_settings:
+  ntfy:
+    topic: alerts
+  headscale:
+    namespace: counter-net

backup.inventory.ini

@@ -0,0 +1,16 @@
+[vps]
+vipy ansible_host=207.154.226.192 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+watchtower ansible_host=206.189.63.167 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+spacey ansible_host=165.232.73.4 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+
+[nodito_host]
+nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ansible_ssh_pass=noesfacilvivirenunmundocentralizado ansible_ssh_private_key_file=~/.ssh/counterganzua
+
+[nodito_vms]
+memos-box ansible_host=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
+
+
+# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
+# Make sure to adjust the username
+[lapy]
+localhost ansible_connection=local ansible_user=counterweight gpg_recipient=counterweightoperator@protonmail.com gpg_key_id=883EDBAA726BD96C

human_script.md

@@ -0,0 +1,897 @@
+# Personal Infrastructure Setup Guide
+
+This guide walks you through setting up your complete personal infrastructure, layer by layer. Each layer must be completed before moving to the next one.
+
+**Automated Setup:** Each layer has a bash script that handles the setup process. The scripts will:
+- Check prerequisites
+- Prompt for required variables
+- Set up configuration files
+- Execute playbooks
+- Verify completion
+
+## Prerequisites
+
+Before starting:
+- You have a domain name
+- You have VPS accounts ready
+- You have nodito ready with Proxmox installed, ssh key in place
+- You have SSH access to all machines
+- You're running this from your laptop (lapy)
+
+---
+
+## Layer 0: Foundation Setup
+
+**Goal:** Set up your laptop (lapy) as the Ansible control node and configure basic settings.
+
+**Script:** `./scripts/setup_layer_0.sh`
+
+### What This Layer Does:
+1. Creates Python virtual environment
+2. Installs Ansible and required Python packages
+3. Installs Ansible Galaxy collections
+4. Guides you through creating `inventory.ini` with your machine IPs
+5. Guides you through creating `infra_vars.yml` with your domain
+6. Creates `services_config.yml` with centralized subdomain settings
+7. Creates `infra_secrets.yml` template for Uptime Kuma credentials
+8. Validates SSH keys exist
+9. Verifies everything is ready for Layer 1
+
+### Required Information:
+- Your domain name (e.g., `contrapeso.xyz`)
+- SSH key path (default: `~/.ssh/counterganzua`)
+- IP addresses for your infrastructure:
+  - vipy (main VPS)
+  - watchtower (monitoring VPS)
+  - spacey (headscale VPS)
+  - nodito (Proxmox server) - optional
+  - **Note:** VMs (like memos-box) will be created later on Proxmox and added to the `nodito_vms` group
+
+### Manual Steps:
+After running the script, you'll need to:
+1. Ensure your SSH key is added to all VPS root users (usually done by VPS provider)
+2. Ensure DNS is configured for your domain (nameservers pointing to your DNS provider)
+
+### Centralized Configuration:
+
+The script creates `ansible/services_config.yml` which contains all service subdomains in one place:
+- Easy to review all subdomains at a glance
+- No need to edit multiple vars files
+- Consistent Caddy settings across all services
+- **Edit this file to customize your subdomains before deploying services**
+
+### Verification:
+The script will verify:
+- ✓ Python venv exists and activated
+- ✓ Ansible installed
+- ✓ Required Python packages installed
+- ✓ Ansible Galaxy collections installed
+- ✓ `inventory.ini` exists and formatted correctly
+- ✓ `infra_vars.yml` exists with domain configured
+- ✓ `services_config.yml` created with subdomain settings
+- ✓ `infra_secrets.yml` template created
+- ✓ SSH key file exists
+
+### Run the Script:
+```bash
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_0.sh
+```
+
+---
+
+## Layer 1A: VPS Basic Setup
+
+**Goal:** Configure users, SSH access, firewall, and fail2ban on VPS machines.
+
+**Script:** `./scripts/setup_layer_1a_vps.sh`
+
+**Can be run independently** - doesn't require Nodito setup.
+
+### What This Layer Does:
+
+For VPSs (vipy, watchtower, spacey):
+1. Creates the `counterweight` user with sudo access
+2. Configures SSH key authentication
+3. Disables root login (by design for security)
+4. Sets up UFW firewall with SSH access
+5. Installs and configures fail2ban
+6. Installs and configures auditd for security logging
+
+### Prerequisites:
+- ✅ Layer 0 complete
+- ✅ SSH key added to all VPS root users
+- ✅ Root access to VPSs
+
+### Verification:
+The script will verify:
+- ✓ Can SSH to all VPSs as root
+- ✓ VPS playbooks complete successfully
+- ✓ Can SSH to all VPSs as `counterweight` user
+- ✓ Firewall is active and configured
+- ✓ fail2ban is running
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_1a_vps.sh
+```
+
+**Note:** After this layer, you will no longer be able to SSH as root to VPSs (by design for security).
+
+---
+
+## Layer 1B: Nodito (Proxmox) Setup
+
+**Goal:** Configure the Nodito Proxmox server.
+
+**Script:** `./scripts/setup_layer_1b_nodito.sh`
+
+**Can be run independently** - doesn't require VPS setup.
+
+### What This Layer Does:
+
+For Nodito (Proxmox server):
+1. Bootstraps SSH key access for root
+2. Creates the `counterweight` user
+3. Updates and secures the system
+4. Disables root login and password authentication
+5. Switches to Proxmox community repositories
+6. Optionally sets up ZFS storage pool (if disks configured)
+7. Optionally creates Debian cloud template
+
+### Prerequisites:
+- ✅ Layer 0 complete
+- ✅ Root password for nodito
+- ✅ Nodito configured in inventory.ini
+
+### Optional: ZFS Setup
+For ZFS storage pool (optional):
+1. SSH into nodito: `ssh root@<nodito-ip>`
+2. List disk IDs: `ls -la /dev/disk/by-id/ | grep -E "(ata-|scsi-|nvme-)"`
+3. Note the disk IDs you want to use
+4. The script will help you create `ansible/infra/nodito/nodito_vars.yml` with disk configuration
+
+⚠️ **Warning:** ZFS setup will DESTROY ALL DATA on specified disks!
+
+### Verification:
+The script will verify:
+- ✓ Nodito bootstrap successful
+- ✓ Community repos configured
+- ✓ Can SSH to nodito as `counterweight` user
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_1b_nodito.sh
+```
+
+**Note:** After this layer, you will no longer be able to SSH as root to nodito (by design for security).
+
+---
+
+## Layer 2: General Infrastructure Tools
+
+**Goal:** Install common utilities needed by various services.
+
+**Script:** `./scripts/setup_layer_2.sh`
+
+### What This Layer Does:
+
+Installs essential tools on machines that need them:
+
+#### rsync
+- **Purpose:** Required for backup operations
+- **Deployed to:** vipy, watchtower, lapy (and optionally other hosts)
+- **Playbook:** `infra/900_install_rsync.yml`
+
+#### Docker + Docker Compose
+- **Purpose:** Required for containerized services
+- **Deployed to:** vipy, watchtower (and optionally other hosts)
+- **Playbook:** `infra/910_docker_playbook.yml`
+
+### Prerequisites:
+- ✅ Layer 0 complete
+- ✅ Layer 1A complete (for VPSs) OR Layer 1B complete (for nodito)
+- ✅ SSH access as counterweight user
+
+### Services That Need These Tools:
+- **rsync:** All backup operations (Uptime Kuma, Vaultwarden, LNBits, etc.)
+- **docker:** Uptime Kuma, Vaultwarden, ntfy-emergency-app
+
+### Verification:
+The script will verify:
+- ✓ rsync installed on specified hosts
+- ✓ Docker and Docker Compose installed on specified hosts
+- ✓ counterweight user added to docker group
+- ✓ Docker service running
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_2.sh
+```
+
+**Note:** This script is interactive and will let you choose which hosts get which tools.
+
+---
+
+## Layer 3: Reverse Proxy (Caddy)
+
+**Goal:** Deploy Caddy reverse proxy for HTTPS termination and routing.
+
+**Script:** `./scripts/setup_layer_3_caddy.sh`
+
+### What This Layer Does:
+
+Installs and configures Caddy web server on VPS machines:
+- Installs Caddy from official repositories
+- Configures Caddy to listen on ports 80/443
+- Opens firewall ports for HTTP/HTTPS
+- Creates `/etc/caddy/sites-enabled/` directory structure
+- Sets up automatic HTTPS with Let's Encrypt
+
+**Deployed to:** vipy, watchtower, spacey
+
+### Why Caddy is Critical:
+
+Caddy provides:
+- **Automatic HTTPS** - Let's Encrypt certificates with auto-renewal
+- **Reverse proxy** - Routes traffic to backend services
+- **Simple configuration** - Each service adds its own config file
+- **HTTP/2 support** - Modern protocol support
+
+### Prerequisites:
+- ✅ Layer 0 complete
+- ✅ Layer 1A complete (VPS setup)
+- ✅ SSH access as counterweight user
+- ✅ Ports 80/443 available on VPSs
+
+### Services That Need Caddy:
+All web services depend on Caddy:
+- Uptime Kuma (watchtower)
+- ntfy (watchtower)
+- Headscale (spacey)
+- Vaultwarden (vipy)
+- Forgejo (vipy)
+- LNBits (vipy)
+- ntfy-emergency-app (vipy)
+
+### Verification:
+The script will verify:
+- ✓ Caddy installed on all target hosts
+- ✓ Caddy service running
+- ✓ Ports 80/443 open in firewall
+- ✓ Sites-enabled directory created
+- ✓ Can reach Caddy default page
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_3_caddy.sh
+```
+
+**Note:** Caddy starts with an empty configuration. Services will add their own config files in later layers.
+
+---
+
+## Layer 4: Core Monitoring & Notifications
+
+**Goal:** Deploy ntfy (notifications) and Uptime Kuma (monitoring platform).
+
+**Script:** `./scripts/setup_layer_4_monitoring.sh`
+
+### What This Layer Does:
+
+Deploys core monitoring infrastructure on watchtower:
+
+#### 4A: ntfy (Notification Service)
+- Installs ntfy from official repositories
+- Configures ntfy with authentication (deny-all by default)
+- Creates admin user for sending notifications
+- Sets up Caddy reverse proxy
+- **Deployed to:** watchtower
+
+#### 4B: Uptime Kuma (Monitoring Platform)
+- Deploys Uptime Kuma via Docker
+- Configures Caddy reverse proxy
+- Sets up data persistence
+- Optionally sets up backup to lapy
+- **Deployed to:** watchtower
+
+### Prerequisites (Complete BEFORE Running):
+
+**1. Previous layers complete:**
+- ✅ Layer 0, 1A, 2, 3 complete (watchtower must be fully set up)
+- ✅ Docker installed on watchtower (from Layer 2)
+- ✅ Caddy running on watchtower (from Layer 3)
+
+**2. Configure subdomains (in centralized config):**
+- ✅ Edit `ansible/services_config.yml` and customize subdomains under `subdomains:` section
+  - Set `ntfy:` to your preferred subdomain (e.g., `ntfy` or `notify`)
+  - Set `uptime_kuma:` to your preferred subdomain (e.g., `uptime` or `kuma`)
+
+**3. Create DNS records that match your configured subdomains:**
+- ✅ Create A record: `<ntfy_subdomain>.<yourdomain>` → watchtower IP
+- ✅ Create A record: `<uptime_kuma_subdomain>.<yourdomain>` → watchtower IP
+- ✅ Wait for DNS propagation (can take minutes to hours)
+- ✅ Verify with: `dig <subdomain>.<yourdomain>` should return watchtower IP
+
+**4. Prepare ntfy admin credentials:**
+- ✅ Decide on username (default: `admin`)
+- ✅ Decide on a secure password (script will prompt you)
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_4_monitoring.sh
+```
+
+The script will prompt you for ntfy admin credentials during deployment.
+
+### Post-Deployment Steps (Complete AFTER Running):
+
+**The script will guide you through most of these, but here's what happens:**
+
+#### Step 1: Set Up Uptime Kuma Admin Account (Manual)
+1. Open browser and visit: `https://<uptime_kuma_subdomain>.<yourdomain>`
+2. On first visit, you'll see the setup page
+3. Create admin username and password
+4. Save these credentials securely
+
+#### Step 2: Update infra_secrets.yml (Manual)
+1. Edit `ansible/infra_secrets.yml`
+2. Add your Uptime Kuma credentials:
+   ```yaml
+   uptime_kuma_username: "your-admin-username"
+   uptime_kuma_password: "your-admin-password"
+   ```
+3. Save the file
+4. **This is required for automated ntfy setup and Layer 6**
+
+#### Step 3: Configure ntfy Notification (Automated)
+**The script will offer to do this automatically!** If you completed Steps 1 & 2, the script will:
+- Connect to Uptime Kuma via API
+- Create ntfy notification configuration
+- Test the connection
+- No manual UI configuration needed!
+
+**Alternatively (Manual):**
+1. In Uptime Kuma web UI, go to **Settings** → **Notifications**
+2. Click **Setup Notification**, choose **ntfy**
+3. Configure with your ntfy subdomain and credentials
+
+#### Step 4: Final Verification (Automated)
+**The script will automatically verify:**
+- ✓ Uptime Kuma credentials in infra_secrets.yml
+- ✓ Can connect to Uptime Kuma API
+- ✓ ntfy notification is configured
+- ✓ All post-deployment steps complete
+
+If anything is missing, the script will tell you exactly what to do!
+
+#### Step 5: Subscribe to Notifications on Your Phone (Optional - Manual)
+1. Install ntfy app: https://github.com/binwiederhier/ntfy-android
+2. Add subscription:
+   - Server: `https://<ntfy_subdomain>.<yourdomain>`
+   - Topic: `alerts` (same as configured in Uptime Kuma)
+   - Username: Your ntfy admin username
+   - Password: Your ntfy admin password
+3. You'll now receive push notifications for all alerts!
+
+**Pro tip:** Run the script again after completing Steps 1 & 2, and it will automatically configure ntfy and verify everything!
+
+### Verification:
+The script will automatically verify:
+- ✓ DNS records are configured correctly (using `dig`)
+- ✓ ntfy service running
+- ✓ Uptime Kuma container running
+- ✓ Caddy configs created for both services
+
+After post-deployment steps, you can test:
+- Visit `https://<ntfy_subdomain>.<yourdomain>` (should load ntfy web UI)
+- Visit `https://<uptime_kuma_subdomain>.<yourdomain>` (should load Uptime Kuma)
+- Send test notification in Uptime Kuma
+
+**Note:** DNS validation requires `dig` command. If not available, validation will be skipped (you can continue but SSL may fail).
+
+### Why This Layer is Critical:
+- **All infrastructure monitoring** (Layer 6) depends on Uptime Kuma
+- **All alerts** go through ntfy
+- Services availability monitoring needs Uptime Kuma
+- Without this layer, you won't know when things break!
+
+---
+
+## Layer 5: VPN Infrastructure (Headscale)
+
+**Goal:** Deploy Headscale for secure mesh networking (like Tailscale, but self-hosted).
+
+**Script:** `./scripts/setup_layer_5_headscale.sh`
+
+**This layer is OPTIONAL** - Skip to Layer 6 if you don't need VPN mesh networking.
+
+### What This Layer Does:
+
+Deploys Headscale coordination server and optionally joins machines to the mesh:
+
+#### 5A: Deploy Headscale Server
+- Installs Headscale on spacey
+- Configures with deny-all ACL policy (you customize later)
+- Creates namespace/user for your network
+- Sets up Caddy reverse proxy
+- Configures embedded DERP server for NAT traversal
+- **Deployed to:** spacey
+
+#### 5B: Join Machines to Mesh (Optional)
+- Installs Tailscale client on target machines
+- Generates ephemeral pre-auth keys
+- Automatically joins machines to your mesh
+- Enables Magic DNS
+- **Can join:** vipy, watchtower, nodito, lapy, etc.
+
+### Prerequisites (Complete BEFORE Running):
+
+**1. Previous layers complete:**
+- ✅ Layer 0, 1A, 3 complete (spacey must be set up)
+- ✅ Caddy running on spacey (from Layer 3)
+
+**2. Configure subdomain (in centralized config):**
+- ✅ Edit `ansible/services_config.yml` and customize `headscale:` under `subdomains:` section (e.g., `headscale` or `vpn`)
+
+**3. Create DNS record that matches your configured subdomain:**
+- ✅ Create A record: `<headscale_subdomain>.<yourdomain>` → spacey IP
+- ✅ Wait for DNS propagation
+- ✅ Verify with: `dig <subdomain>.<yourdomain>` should return spacey IP
+
+**4. Decide on namespace name:**
+- ✅ Choose a namespace for your network (default: `counter-net`)
+- ✅ This is set in `headscale_vars.yml` as `headscale_namespace`
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_5_headscale.sh
+```
+
+The script will:
+1. Validate DNS configuration
+2. Deploy Headscale server
+3. Offer to join machines to the mesh
+
+### Post-Deployment Steps:
+
+#### Configure ACL Policies (Required for machines to communicate)
+1. SSH into spacey: `ssh counterweight@<spacey-ip>`
+2. Edit ACL file: `sudo nano /etc/headscale/acl.json`
+3. Configure rules (example - allow all):
+   ```json
+   {
+     "ACLs": [
+       {"action": "accept", "src": ["*"], "dst": ["*:*"]}
+     ]
+   }
+   ```
+4. Restart Headscale: `sudo systemctl restart headscale`
+
+**Default is deny-all for security** - you must configure ACLs for machines to talk!
+
+#### Join Additional Machines Manually
+For machines not in inventory (mobile, desktop):
+1. Install Tailscale client on device
+2. Generate pre-auth key on spacey:
+   ```bash
+   ssh counterweight@<spacey-ip>
+   sudo headscale preauthkeys create --user <namespace> --reusable
+   ```
+3. Connect using your Headscale server:
+   ```bash
+   tailscale up --login-server https://<headscale_subdomain>.<yourdomain> --authkey <key>
+   ```
+
+### Automatic Uptime Kuma Monitor:
+
+**The playbook will automatically create a monitor in Uptime Kuma:**
+- ✅ **Headscale** - monitors `https://<subdomain>/health`
+- Added to "services" monitor group
+- Uses ntfy notification (if configured)
+- Check every 60 seconds
+
+**Prerequisites:** Uptime Kuma credentials must be in `infra_secrets.yml` (from Layer 4)
+
+### Verification:
+The script will automatically verify:
+- ✓ DNS records configured correctly
+- ✓ Headscale installed and running
+- ✓ Namespace created
+- ✓ Caddy config created
+- ✓ Machines joined (if selected)
+- ✓ Monitor created in Uptime Kuma "services" group
+
+List connected devices:
+```bash
+ssh counterweight@<spacey-ip>
+sudo headscale nodes list
+```
+
+### Why Use Headscale:
+- **Secure communication** between all your machines
+- **Magic DNS** - access machines by hostname
+- **NAT traversal** - works even behind firewalls
+- **Self-hosted** - full control of your VPN
+- **Mobile support** - use official Tailscale apps
+
+### Backup:
+Optional backup to lapy:
+```bash
+ansible-playbook -i inventory.ini services/headscale/setup_backup_headscale_to_lapy.yml
+```
+
+---
+
+## Layer 6: Infrastructure Monitoring
+
+**Goal:** Deploy automated monitoring for disk usage, system health, and CPU temperature.
+
+**Script:** `./scripts/setup_layer_6_infra_monitoring.sh`
+
+### What This Layer Does:
+
+Deploys monitoring scripts that report to Uptime Kuma:
+
+#### 6A: Disk Usage Monitoring
+- Monitors disk usage on specified mount points
+- Sends alerts when usage exceeds threshold (default: 80%)
+- Creates Uptime Kuma push monitors automatically
+- Organizes monitors in host-specific groups
+- **Deploys to:** All hosts (selectable)
+
+#### 6B: System Healthcheck
+- Sends regular heartbeat pings to Uptime Kuma
+- Alerts if system stops responding
+- "No news is good news" monitoring
+- **Deploys to:** All hosts (selectable)
+
+#### 6C: CPU Temperature Monitoring (Nodito only)
+- Monitors CPU temperature on Proxmox server
+- Alerts when temperature exceeds threshold (default: 80°C)
+- **Deploys to:** nodito (if configured)
+
+### Prerequisites (Complete BEFORE Running):
+
+**1. Previous layers complete:**
+- ✅ Layer 0, 1A/1B, 4 complete
+- ✅ Uptime Kuma deployed and configured (Layer 4)
+- ✅ **CRITICAL:** `infra_secrets.yml` has Uptime Kuma credentials
+
+**2. Uptime Kuma API credentials ready:**
+- ✅ Must have completed Layer 4 post-deployment steps
+- ✅ `ansible/infra_secrets.yml` must contain:
+  ```yaml
+  uptime_kuma_username: "your-username"
+  uptime_kuma_password: "your-password"
+  ```
+
+**3. Python dependencies installed:**
+- ✅ `uptime-kuma-api` must be in requirements.txt
+- ✅ Should already be installed from Layer 0
+- ✅ Verify: `pip list | grep uptime-kuma-api`
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_6_infra_monitoring.sh
+```
+
+The script will:
+1. Verify Uptime Kuma credentials
+2. Offer to deploy disk usage monitoring
+3. Offer to deploy system healthchecks
+4. Offer to deploy CPU temp monitoring (nodito only)
+5. Test monitor creation and alerts
+
+### What Gets Deployed:
+
+**For each monitored host:**
+- Push monitor in Uptime Kuma (upside-down mode)
+- Monitor group named `{hostname} - infra`
+- Systemd service for monitoring script
+- Systemd timer for periodic execution
+- Log file for monitoring history
+
+**Default settings (customizable):**
+- Disk usage threshold: 80%
+- Disk check interval: 15 minutes
+- Healthcheck interval: 60 seconds
+- CPU temp threshold: 80°C
+- Monitored mount point: `/` (root)
+
+### Customization Options:
+
+Change thresholds and intervals:
+```bash
+# Disk monitoring with custom settings
+ansible-playbook -i inventory.ini infra/410_disk_usage_alerts.yml \
+  -e "disk_usage_threshold_percent=85" \
+  -e "disk_check_interval_minutes=10" \
+  -e "monitored_mount_point=/home"
+
+# Healthcheck with custom interval
+ansible-playbook -i inventory.ini infra/420_system_healthcheck.yml \
+  -e "healthcheck_interval_seconds=30"
+
+# CPU temp with custom threshold
+ansible-playbook -i inventory.ini infra/430_cpu_temp_alerts.yml \
+  -e "temp_threshold_celsius=75"
+```
+
+### Verification:
+The script will automatically verify:
+- ✓ Uptime Kuma API accessible
+- ✓ Monitors created in Uptime Kuma
+- ✓ Monitor groups created
+- ✓ Systemd services running
+- ✓ Can send test alerts
+
+Check Uptime Kuma web UI:
+- Monitors should appear organized by host
+- Should receive test pings
+- Alerts will show when thresholds exceeded
+
+### Post-Deployment:
+
+**Monitor your infrastructure:**
+1. Open Uptime Kuma web UI
+2. See all monitors organized by host groups
+3. Configure notification rules per monitor
+4. Set up status pages (optional)
+
+**Test alerts:**
+```bash
+# Trigger disk usage alert (fill disk temporarily)
+# Trigger healthcheck alert (stop the service)
+# Check ntfy for notifications
+```
+
+### Why This Layer is Important:
+- **Proactive monitoring** - Know about issues before users do
+- **Disk space alerts** - Prevent services from failing
+- **System health** - Detect crashed/frozen machines
+- **Temperature monitoring** - Prevent hardware damage
+- **Organized** - All monitors grouped by host
+
+---
+
+## Layer 7: Core Services
+
+**Goal:** Deploy core applications: Vaultwarden, Forgejo, and LNBits.
+
+**Script:** `./scripts/setup_layer_7_services.sh`
+
+### What This Layer Does:
+
+Deploys main services on vipy:
+
+#### 7A: Vaultwarden (Password Manager)
+- Deploys via Docker
+- Configures Caddy reverse proxy
+- Sets up fail2ban protection
+- Enables sign-ups initially (disable after creating first user)
+- **Deployed to:** vipy
+
+#### 7B: Forgejo (Git Server)
+- Installs Forgejo binary
+- Creates git user and directories
+- Configures Caddy reverse proxy
+- Enables SSH cloning
+- **Deployed to:** vipy
+
+#### 7C: LNBits (Lightning Wallet)
+- Installs system dependencies and uv (Python 3.12 tooling)
+- Clones LNBits version v1.3.1
+- Syncs dependencies with uv targeting Python 3.12
+- Configures with FakeWallet backend (for testing)
+- Creates systemd service
+- Configures Caddy reverse proxy
+- **Deployed to:** vipy
+
+### Prerequisites (Complete BEFORE Running):
+
+**1. Previous layers complete:**
+- ✅ Layer 0, 1A, 2, 3 complete
+- ✅ Docker installed on vipy (Layer 2)
+- ✅ Caddy running on vipy (Layer 3)
+
+**2. Configure subdomains (in centralized config):**
+- ✅ Edit `ansible/services_config.yml` and customize subdomains under `subdomains:` section:
+  - Set `vaultwarden:` to your preferred subdomain (e.g., `vault` or `passwords`)
+  - Set `forgejo:` to your preferred subdomain (e.g., `git` or `code`)
+  - Set `lnbits:` to your preferred subdomain (e.g., `lnbits` or `wallet`)
+
+**3. Create DNS records matching your subdomains:**
+- ✅ Create A record: `<vaultwarden_subdomain>.<yourdomain>` → vipy IP
+- ✅ Create A record: `<forgejo_subdomain>.<yourdomain>` → vipy IP
+- ✅ Create A record: `<lnbits_subdomain>.<yourdomain>` → vipy IP
+- ✅ Wait for DNS propagation
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_7_services.sh
+```
+
+The script will:
+1. Validate DNS configuration
+2. Offer to deploy each service
+3. Configure backups (optional)
+
+### Post-Deployment Steps:
+
+#### Vaultwarden:
+1. Visit `https://<vaultwarden_subdomain>.<yourdomain>`
+2. Create your first user account
+3. **Important:** Disable sign-ups after first user:
+   ```bash
+   ansible-playbook -i inventory.ini services/vaultwarden/disable_vaultwarden_sign_ups_playbook.yml
+   ```
+4. Optional: Set up backup to lapy
+
+#### Forgejo:
+1. Visit `https://<forgejo_subdomain>.<yourdomain>`
+2. Create admin account on first visit
+3. Default: registrations disabled for security
+4. SSH cloning works automatically after adding SSH key
+
+#### LNBits:
+1. Visit `https://<lnbits_subdomain>.<yourdomain>`
+2. Create superuser on first visit
+3. **Important:** Default uses FakeWallet (testing only)
+4. Configure real Lightning backend:
+   - Edit `/opt/lnbits/lnbits/.env` on vipy
+   - Or use the superuser UI to configure backend
+5. Disable new user registration for security
+6. Optional: Set up encrypted backup to lapy
+
+### Backup Configuration:
+
+After services are stable, set up backups:
+
+**Vaultwarden backup:**
+```bash
+ansible-playbook -i inventory.ini services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml
+```
+
+**LNBits backup (GPG encrypted):**
+```bash
+ansible-playbook -i inventory.ini services/lnbits/setup_backup_lnbits_to_lapy.yml
+```
+
+**Note:** Forgejo backups are not automated - backup manually or set up your own solution.
+
+### Automatic Uptime Kuma Monitors:
+
+**The playbooks will automatically create monitors in Uptime Kuma for each service:**
+- ✅ **Vaultwarden** - monitors `https://<subdomain>/alive`
+- ✅ **Forgejo** - monitors `https://<subdomain>/api/healthz`
+- ✅ **LNBits** - monitors `https://<subdomain>/api/v1/health`
+
+All monitors:
+- Added to "services" monitor group
+- Use ntfy notification (if configured)
+- Check every 60 seconds
+- 3 retries before alerting
+
+**Prerequisites:** Uptime Kuma credentials must be in `infra_secrets.yml` (from Layer 4)
+
+### Verification:
+The script will automatically verify:
+- ✓ DNS records configured
+- ✓ Services deployed
+- ✓ Docker containers running (Vaultwarden)
+- ✓ Systemd services running (Forgejo, LNBits)
+- ✓ Caddy configs created
+
+Manual verification:
+- Visit each service's subdomain
+- Create admin/first user accounts
+- Test functionality
+- Check Uptime Kuma for new monitors in "services" group
+
+### Why These Services:
+- **Vaultwarden** - Self-hosted password manager (Bitwarden compatible)
+- **Forgejo** - Self-hosted Git server (GitHub/GitLab alternative)
+- **LNBits** - Lightning Network wallet and accounts system
+
+---
+
+## Layer 8: Secondary Services
+
+**Goal:** Deploy auxiliary services that depend on the core stack: ntfy-emergency-app and memos.
+
+**Script:** `./scripts/setup_layer_8_secondary_services.sh`
+
+### What This Layer Does:
+- Deploys the ntfy-emergency-app container on vipy and proxies it through Caddy
+- Optionally deploys Memos on `memos-box` (skips automatically if the host is not yet in `inventory.ini`)
+
+### Prerequisites (Complete BEFORE Running):
+- ✅ Layers 0–7 complete (Caddy, ntfy, and Uptime Kuma already online)
+- ✅ `ansible/services_config.yml` reviewed so the `ntfy_emergency_app` and `memos` subdomains match your plan
+- ✅ `ansible/infra_secrets.yml` contains valid `ntfy_username` and `ntfy_password`
+- ✅ DNS A records created for the subdomains (see below)
+- ✅ If deploying Memos, ensure `memos-box` exists in `inventory.ini` and is reachable as the `counterweight` user
+
+### DNS Requirements:
+- `<ntfy_emergency_app>.<domain>` → vipy IP
+- `<memos>.<domain>` → memos-box IP (skip if memos not yet provisioned)
+
+The script runs `dig` to validate DNS before deploying and will warn if records are missing or pointing elsewhere.
+
+### Run the Script:
+```bash
+source venv/bin/activate
+cd /home/counterweight/personal_infra
+./scripts/setup_layer_8_secondary_services.sh
+```
+
+You can deploy each service independently; the script asks for confirmation before running each playbook.
+
+### Post-Deployment Steps:
+- **ntfy-emergency-app:** Visit the emergency subdomain, trigger a test notification, and verify ntfy receives it
+- **Memos (if deployed):** Visit the memos subdomain, create the first admin user, and adjust settings from the UI
+
+### Verification:
+- The script checks for the presence of Caddy configs, running containers, and Memos systemd service status
+- Review Uptime Kuma or add monitors for these services if you want automatic alerting
+
+### Optional Follow-Ups:
+- Configure backups for any new data stores (e.g., snapshot memos data)
+- Add Uptime Kuma monitors for the new services if you want automated alerting
+
+---
+
+## Troubleshooting
+
+### Common Issues
+
+#### SSH Connection Fails
+- Verify VPS is running and accessible
+- Check SSH key is in the correct location
+- Ensure SSH key has correct permissions (600)
+- Try manual SSH: `ssh -i ~/.ssh/counterganzua root@<ip>`
+
+#### Ansible Not Found
+- Make sure you've activated the venv: `source venv/bin/activate`
+- Run Layer 0 script again
+
+#### DNS Not Resolving
+- DNS changes can take up to 24-48 hours to propagate
+- Use `dig <subdomain>.<domain>` to check DNS status
+- You can proceed with setup; services will work once DNS propagates
+
+---
+
+## Progress Tracking
+
+Use this checklist to track your progress:
+
+- [ ] Layer 0: Foundation Setup
+- [ ] Layer 1A: VPS Basic Setup
+- [ ] Layer 1B: Nodito (Proxmox) Setup
+- [ ] Layer 2: General Infrastructure Tools
+- [ ] Layer 3: Reverse Proxy (Caddy)
+- [ ] Layer 4: Core Monitoring & Notifications
+- [ ] Layer 5: VPN Infrastructure (Headscale)
+- [ ] Layer 6: Infrastructure Monitoring
+- [ ] Layer 7: Core Services
+- [ ] Layer 8: Secondary Services
+- [ ] Backups Configured
+

requirements.txt

@@ -8,3 +8,4 @@ packaging==25.0
 pycparser==2.22
 PyYAML==6.0.2
 resolvelib==1.0.1
+uptime-kuma-api>=1.2.1

scripts/setup_layer_0.sh

@@ -0,0 +1,488 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 0: Foundation Setup
+# 
+# This script sets up your laptop (lapy) as the Ansible control node.
+# It prepares all the prerequisites needed for the infrastructure deployment.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+prompt_user() {
+    local prompt="$1"
+    local default="$2"
+    local result
+    
+    if [ -n "$default" ]; then
+        read -p "$(echo -e ${BLUE}${prompt}${NC} [${default}]: )" result
+        result="${result:-$default}"
+    else
+        read -p "$(echo -e ${BLUE}${prompt}${NC}: )" result
+    fi
+    
+    echo "$result"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+###############################################################################
+# Main Setup Functions
+###############################################################################
+
+check_prerequisites() {
+    print_header "Checking Prerequisites"
+    
+    # Check if we're in the right directory
+    if [ ! -f "$PROJECT_ROOT/README.md" ] || [ ! -d "$PROJECT_ROOT/ansible" ]; then
+        print_error "Not in the correct project directory"
+        echo "Expected: $PROJECT_ROOT"
+        exit 1
+    fi
+    print_success "Running from correct directory: $PROJECT_ROOT"
+    
+    # Check if Python 3 is installed
+    if ! command -v python3 &> /dev/null; then
+        print_error "Python 3 is not installed. Please install Python 3 first."
+        exit 1
+    fi
+    print_success "Python 3 found: $(python3 --version)"
+    
+    # Check if git is installed
+    if ! command -v git &> /dev/null; then
+        print_warning "Git is not installed. Some features may not work."
+    else
+        print_success "Git found: $(git --version | head -n1)"
+    fi
+}
+
+setup_python_venv() {
+    print_header "Setting Up Python Virtual Environment"
+    
+    cd "$PROJECT_ROOT"
+    
+    if [ -d "venv" ]; then
+        print_info "Virtual environment already exists"
+        if confirm_action "Recreate virtual environment?"; then
+            rm -rf venv
+            python3 -m venv venv
+            print_success "Virtual environment recreated"
+        else
+            print_success "Using existing virtual environment"
+        fi
+    else
+        python3 -m venv venv
+        print_success "Virtual environment created"
+    fi
+    
+    # Activate venv
+    source venv/bin/activate
+    print_success "Virtual environment activated"
+    
+    # Upgrade pip
+    print_info "Upgrading pip..."
+    pip install --upgrade pip > /dev/null 2>&1
+    print_success "pip upgraded"
+}
+
+install_python_requirements() {
+    print_header "Installing Python Requirements"
+    
+    cd "$PROJECT_ROOT"
+    
+    if [ ! -f "requirements.txt" ]; then
+        print_error "requirements.txt not found"
+        exit 1
+    fi
+    
+    print_info "Installing packages from requirements.txt..."
+    pip install -r requirements.txt
+    print_success "Python requirements installed"
+    
+    # Verify Ansible installation
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible installation failed"
+        exit 1
+    fi
+    print_success "Ansible installed: $(ansible --version | head -n1)"
+}
+
+install_ansible_collections() {
+    print_header "Installing Ansible Galaxy Collections"
+    
+    cd "$PROJECT_ROOT/ansible"
+    
+    if [ ! -f "requirements.yml" ]; then
+        print_warning "requirements.yml not found, skipping Ansible collections"
+        return
+    fi
+    
+    print_info "Installing Ansible Galaxy collections..."
+    ansible-galaxy collection install -r requirements.yml
+    print_success "Ansible Galaxy collections installed"
+}
+
+setup_inventory_file() {
+    print_header "Setting Up Inventory File"
+    
+    cd "$PROJECT_ROOT/ansible"
+    
+    if [ -f "inventory.ini" ]; then
+        print_info "inventory.ini already exists"
+        cat inventory.ini
+        echo ""
+        if ! confirm_action "Do you want to update it?"; then
+            print_success "Using existing inventory.ini"
+            return
+        fi
+    fi
+    
+    print_info "Let's configure your infrastructure hosts"
+    echo ""
+    
+    # Collect information
+    echo -e -n "${BLUE}SSH key path${NC} [~/.ssh/counterganzua]: "
+    read ssh_key
+    ssh_key="${ssh_key:-~/.ssh/counterganzua}"
+    
+    echo ""
+    echo "Enter the IP addresses for your infrastructure (VMs will be added later):"
+    echo ""
+    
+    echo -e -n "${BLUE}vipy${NC} (main VPS) IP: "
+    read vipy_ip
+    echo -e -n "${BLUE}watchtower${NC} (monitoring VPS) IP: "
+    read watchtower_ip
+    echo -e -n "${BLUE}spacey${NC} (headscale VPS) IP: "
+    read spacey_ip
+    echo -e -n "${BLUE}nodito${NC} (Proxmox server) IP [optional]: "
+    read nodito_ip
+    
+    echo ""
+    echo -e -n "${BLUE}Your username on lapy${NC} [$(whoami)]: "
+    read lapy_user
+    lapy_user="${lapy_user:-$(whoami)}"
+    
+    echo -e -n "${BLUE}GPG recipient email${NC} [optional, for encrypted backups]: "
+    read gpg_email
+    echo -e -n "${BLUE}GPG key ID${NC} [optional, for encrypted backups]: "
+    read gpg_key
+    
+    # Generate inventory.ini
+    cat > inventory.ini << EOF
+# Ansible Inventory File
+# Generated by setup_layer_0.sh
+
+EOF
+    
+    vps_entries=""
+    if [ -n "$vipy_ip" ]; then
+        vps_entries+="vipy ansible_host=$vipy_ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=$ssh_key\n"
+    fi
+    if [ -n "$watchtower_ip" ]; then
+        vps_entries+="watchtower ansible_host=$watchtower_ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=$ssh_key\n"
+    fi
+    if [ -n "$spacey_ip" ]; then
+        vps_entries+="spacey ansible_host=$spacey_ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=$ssh_key\n"
+    fi
+    
+    if [ -n "$vps_entries" ]; then
+        cat >> inventory.ini << EOF
+[vps]
+${vps_entries}
+EOF
+    fi
+    
+    if [ -n "$nodito_ip" ]; then
+        cat >> inventory.ini << EOF
+[nodito_host]
+nodito ansible_host=$nodito_ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=$ssh_key
+
+EOF
+    fi
+    
+    # Add nodito_vms placeholder for VMs that will be created later
+    cat >> inventory.ini << EOF
+# Nodito VMs - These don't exist yet and will be created on the Proxmox server
+# Add them here once you create VMs on nodito (e.g., memos-box, etc.)
+[nodito_vms]
+# Example:
+# memos_box ansible_host=192.168.1.150 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=$ssh_key
+
+EOF
+    
+    # Add lapy
+    cat >> inventory.ini << EOF
+# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
+[lapy]
+localhost ansible_connection=local ansible_user=$lapy_user
+EOF
+    
+    if [ -n "$gpg_email" ] && [ -n "$gpg_key" ]; then
+        echo " gpg_recipient=$gpg_email gpg_key_id=$gpg_key" >> inventory.ini
+    fi
+    
+    print_success "inventory.ini created"
+    echo ""
+    print_info "Review your inventory file:"
+    cat inventory.ini
+    echo ""
+}
+
+setup_infra_vars() {
+    print_header "Setting Up Infrastructure Variables"
+    
+    cd "$PROJECT_ROOT/ansible"
+    
+    if [ -f "infra_vars.yml" ]; then
+        print_info "infra_vars.yml already exists"
+        cat infra_vars.yml
+        echo ""
+        if ! confirm_action "Do you want to update it?"; then
+            print_success "Using existing infra_vars.yml"
+            return
+        fi
+    fi
+    
+    echo ""
+    echo -e -n "${BLUE}Your root domain${NC} (e.g., contrapeso.xyz): "
+    read domain
+    
+    while [ -z "$domain" ]; do
+        print_warning "Domain cannot be empty"
+        echo -e -n "${BLUE}Your root domain${NC}: "
+        read domain
+    done
+    
+    cat > infra_vars.yml << EOF
+# Infrastructure Variables
+# Generated by setup_layer_0.sh
+
+new_user: counterweight
+ssh_port: 22
+allow_ssh_from: "any"
+root_domain: $domain
+EOF
+    
+    print_success "infra_vars.yml created"
+    echo ""
+    print_info "Contents:"
+    cat infra_vars.yml
+    echo ""
+}
+
+setup_services_config() {
+    print_header "Setting Up Services Configuration"
+    
+    cd "$PROJECT_ROOT/ansible"
+    
+    if [ -f "services_config.yml" ]; then
+        print_info "services_config.yml already exists"
+        if ! confirm_action "Do you want to recreate it from template?"; then
+            print_success "Using existing services_config.yml"
+            return
+        fi
+    fi
+    
+    if [ ! -f "services_config.yml.example" ]; then
+        print_error "services_config.yml.example not found"
+        return
+    fi
+    
+    cp services_config.yml.example services_config.yml
+    
+    print_success "services_config.yml created"
+    echo ""
+    print_info "This file centralizes all service subdomains and Caddy settings"
+    print_info "Customize subdomains in: ansible/services_config.yml"
+    echo ""
+}
+
+setup_infra_secrets() {
+    print_header "Setting Up Infrastructure Secrets"
+    
+    cd "$PROJECT_ROOT/ansible"
+    
+    if [ -f "infra_secrets.yml" ]; then
+        print_warning "infra_secrets.yml already exists"
+        if ! confirm_action "Do you want to recreate the template?"; then
+            print_success "Using existing infra_secrets.yml"
+            return
+        fi
+    fi
+    
+    cat > infra_secrets.yml << EOF
+# Infrastructure Secrets
+# Generated by setup_layer_0.sh
+# 
+# IMPORTANT: This file contains sensitive credentials
+# It is already in .gitignore - DO NOT commit it to git
+#
+# You'll need to fill in the Uptime Kuma credentials after Layer 4
+# when you deploy Uptime Kuma
+
+# Uptime Kuma Credentials (fill these in after deploying Uptime Kuma in Layer 4)
+uptime_kuma_username: ""
+uptime_kuma_password: ""
+EOF
+    
+    print_success "infra_secrets.yml template created"
+    print_warning "You'll need to fill in Uptime Kuma credentials after Layer 4"
+    echo ""
+}
+
+validate_ssh_key() {
+    print_header "Validating SSH Key"
+    
+    cd "$PROJECT_ROOT/ansible"
+    
+    # Extract SSH key path from inventory
+    if [ -f "inventory.ini" ]; then
+        ssh_key=$(grep "ansible_ssh_private_key_file" inventory.ini | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+        
+        # Expand tilde
+        ssh_key="${ssh_key/#\~/$HOME}"
+        
+        if [ -f "$ssh_key" ]; then
+            print_success "SSH key found: $ssh_key"
+            
+            # Check permissions
+            perms=$(stat -c "%a" "$ssh_key" 2>/dev/null || stat -f "%OLp" "$ssh_key" 2>/dev/null)
+            if [ "$perms" != "600" ]; then
+                print_warning "SSH key permissions are $perms (should be 600)"
+                if confirm_action "Fix permissions?"; then
+                    chmod 600 "$ssh_key"
+                    print_success "Permissions fixed"
+                fi
+            else
+                print_success "SSH key permissions are correct (600)"
+            fi
+        else
+            print_error "SSH key not found: $ssh_key"
+            print_warning "Make sure to create your SSH key before proceeding to Layer 1"
+            echo ""
+            echo "To generate a new SSH key:"
+            echo "  ssh-keygen -t ed25519 -f $ssh_key -C \"your-email@example.com\""
+        fi
+    else
+        print_warning "inventory.ini not found, skipping SSH key validation"
+    fi
+}
+
+print_summary() {
+    print_header "Layer 0 Setup Complete! 🎉"
+    
+    echo "Summary of what was configured:"
+    echo ""
+    print_success "Python virtual environment created and activated"
+    print_success "Ansible and dependencies installed"
+    print_success "Ansible Galaxy collections installed"
+    print_success "inventory.ini configured with your hosts"
+    print_success "infra_vars.yml configured with your domain"
+    print_success "services_config.yml created with subdomain settings"
+    print_success "infra_secrets.yml template created"
+    echo ""
+    
+    print_info "Before proceeding to Layer 1:"
+    echo "  1. Ensure your SSH key is added to all VPS root users"
+    echo "  2. Verify you can SSH into each machine manually"
+    echo "  3. Configure DNS nameservers for your domain (if not done)"
+    echo ""
+    
+    print_info "Note about inventory groups:"
+    echo "  • [nodito_vms] group created as placeholder"
+    echo "  • These VMs will be created later on Proxmox"
+    echo "  • Add their host entries to inventory.ini once created"
+    echo ""
+    
+    print_info "To test SSH access to a host:"
+    echo "  ssh -i ~/.ssh/counterganzua root@<host-ip>"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Review the files in ansible/"
+    echo "  2. Test SSH connections to your hosts"
+    echo "  3. Proceed to Layer 1: ./scripts/setup_layer_1.sh"
+    echo ""
+    
+    print_warning "Remember to activate the venv before running other commands:"
+    echo "  source venv/bin/activate"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "🚀 Layer 0: Foundation Setup"
+    
+    echo "This script will set up your laptop (lapy) as the Ansible control node."
+    echo "It will install all prerequisites and configure basic settings."
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 0 setup?"; then
+        echo "Setup cancelled."
+        exit 0
+    fi
+    
+    check_prerequisites
+    setup_python_venv
+    install_python_requirements
+    install_ansible_collections
+    setup_inventory_file
+    setup_infra_vars
+    setup_services_config
+    setup_infra_secrets
+    validate_ssh_key
+    print_summary
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_1a_vps.sh

@@ -0,0 +1,393 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 1A: VPS Basic Setup
+# 
+# This script configures users, SSH, firewall, and fail2ban on VPS machines.
+# Runs independently - can be executed without Nodito setup.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+check_layer_0_complete() {
+    print_header "Verifying Layer 0 Prerequisites"
+    
+    local errors=0
+    
+    # Check if venv exists
+    if [ ! -d "$PROJECT_ROOT/venv" ]; then
+        print_error "Python venv not found. Run Layer 0 first."
+        ((errors++))
+    else
+        print_success "Python venv exists"
+    fi
+    
+    # Check if we're in a venv
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+    
+    # Check if Ansible is installed
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found: $(ansible --version | head -n1)"
+    fi
+    
+    # Check if inventory.ini exists
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+    
+    # Check if infra_vars.yml exists
+    if [ ! -f "$ANSIBLE_DIR/infra_vars.yml" ]; then
+        print_error "infra_vars.yml not found"
+        ((errors++))
+    else
+        print_success "infra_vars.yml exists"
+    fi
+    
+    if [ $errors -gt 0 ]; then
+        print_error "Layer 0 is not complete. Please run ./scripts/setup_layer_0.sh first"
+        exit 1
+    fi
+    
+    print_success "Layer 0 prerequisites verified"
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    
+    # Parse inventory.ini directly - more reliable than ansible-inventory
+    if [ -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        # Look for the group section [target]
+        local in_section=false
+        local hosts=""
+        while IFS= read -r line; do
+            # Remove comments and whitespace
+            line=$(echo "$line" | sed 's/#.*$//' | xargs)
+            [ -z "$line" ] && continue
+            
+            # Check if we're entering the target section
+            if [[ "$line" =~ ^\[$target\]$ ]]; then
+                in_section=true
+                continue
+            fi
+            
+            # Check if we're entering a different section
+            if [[ "$line" =~ ^\[.*\]$ ]]; then
+                in_section=false
+                continue
+            fi
+            
+            # If we're in the target section, extract hostname
+            if [ "$in_section" = true ]; then
+                local hostname=$(echo "$line" | awk '{print $1}')
+                if [ -n "$hostname" ]; then
+                    hosts="$hosts $hostname"
+                fi
+            fi
+        done < "$ANSIBLE_DIR/inventory.ini"
+        echo "$hosts" | xargs
+    fi
+}
+
+check_vps_configured() {
+    print_header "Checking VPS Configuration"
+    
+    # Get all hosts from the vps group
+    local vps_hosts=$(get_hosts_from_inventory "vps")
+    local has_vps=false
+    
+    # Check for expected VPS hostnames
+    for expected_host in vipy watchtower spacey; do
+        if echo "$vps_hosts" | grep -q "\b$expected_host\b"; then
+            print_success "$expected_host configured"
+            has_vps=true
+        else
+            print_info "$expected_host not configured (skipping)"
+        fi
+    done
+    
+    if [ "$has_vps" = false ]; then
+        print_error "No VPSs configured in inventory.ini"
+        print_info "Add at least one VPS (vipy, watchtower, or spacey) to the [vps] group to proceed"
+        exit 1
+    fi
+    
+    echo ""
+}
+
+check_ssh_connectivity() {
+    print_header "Testing SSH Connectivity as Root"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    print_info "Using SSH key: $ssh_key"
+    echo ""
+    
+    local all_good=true
+    
+    # Get all hosts from the vps group
+    local vps_hosts=$(get_hosts_from_inventory "vps")
+    
+    # Test VPSs (vipy, watchtower, spacey)
+    for expected_host in vipy watchtower spacey; do
+        if echo "$vps_hosts" | grep -q "\b$expected_host\b"; then
+            print_info "Testing SSH to $expected_host as root..."
+            if timeout 10 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes root@$expected_host "echo 'SSH OK'" &>/dev/null; then
+                print_success "SSH to $expected_host as root: OK"
+            else
+                print_error "Cannot SSH to $expected_host as root"
+                print_warning "Make sure your SSH key is added to root on $expected_host"
+                all_good=false
+            fi
+        fi
+    done
+    
+    if [ "$all_good" = false ]; then
+        echo ""
+        print_error "SSH connectivity test failed"
+        print_info "To fix this:"
+        echo "  1. Ensure your VPS provider has added your SSH key to root"
+        echo "  2. Test manually: ssh -i $ssh_key root@<host>"
+        echo ""
+        if ! confirm_action "Continue anyway?"; then
+            exit 1
+        fi
+    fi
+    
+    echo ""
+    print_success "SSH connectivity verified"
+}
+
+###############################################################################
+# VPS Setup Functions
+###############################################################################
+
+setup_vps_users_and_access() {
+    print_header "Setting Up Users and SSH Access on VPSs"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Create the 'counterweight' user with sudo access"
+    echo "  • Configure SSH key authentication"
+    echo "  • Disable root login (optional, configured in playbook)"
+    echo ""
+    print_info "Running: ansible-playbook -i inventory.ini infra/01_user_and_access_setup_playbook.yml"
+    echo ""
+    
+    if ! confirm_action "Proceed with user and access setup?"; then
+        print_warning "Skipped user and access setup"
+        return 1
+    fi
+    
+    # Run the playbook with -e 'ansible_user="root"' to use root for this first run
+    if ansible-playbook -i inventory.ini infra/01_user_and_access_setup_playbook.yml -e 'ansible_user="root"'; then
+        print_success "User and access setup complete"
+        return 0
+    else
+        print_error "User and access setup failed"
+        return 1
+    fi
+}
+
+setup_vps_firewall_and_fail2ban() {
+    print_header "Setting Up Firewall and Fail2ban on VPSs"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Configure UFW firewall with SSH access"
+    echo "  • Install and configure fail2ban for brute force protection"
+    echo "  • Install and configure auditd for security logging"
+    echo ""
+    print_info "Running: ansible-playbook -i inventory.ini infra/02_firewall_and_fail2ban_playbook.yml"
+    echo ""
+    
+    if ! confirm_action "Proceed with firewall and fail2ban setup?"; then
+        print_warning "Skipped firewall setup"
+        return 1
+    fi
+    
+    # Now use the default counterweight user
+    if ansible-playbook -i inventory.ini infra/02_firewall_and_fail2ban_playbook.yml; then
+        print_success "Firewall and fail2ban setup complete"
+        return 0
+    else
+        print_error "Firewall setup failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+verify_layer_1a() {
+    print_header "Verifying Layer 1A Completion"
+    
+    cd "$ANSIBLE_DIR"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    # Test SSH as counterweight user
+    print_info "Testing SSH as counterweight user..."
+    echo ""
+    
+    local all_good=true
+    
+    # Get all hosts from the vps group
+    local vps_hosts=$(get_hosts_from_inventory "vps")
+    
+    for expected_host in vipy watchtower spacey; do
+        if echo "$vps_hosts" | grep -q "\b$expected_host\b"; then
+            if timeout 10 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$expected_host "echo 'SSH OK'" &>/dev/null; then
+                print_success "SSH to $expected_host as counterweight: OK"
+            else
+                print_error "Cannot SSH to $expected_host as counterweight"
+                all_good=false
+            fi
+        fi
+    done
+    
+    echo ""
+    if [ "$all_good" = true ]; then
+        print_success "All SSH connectivity verified"
+    else
+        print_warning "Some SSH tests failed - manual verification recommended"
+        print_info "Test manually: ssh -i $ssh_key counterweight@<host>"
+    fi
+}
+
+###############################################################################
+# Summary Functions
+###############################################################################
+
+print_summary() {
+    print_header "Layer 1A: VPS Setup Complete! 🎉"
+    
+    echo "Summary of what was configured:"
+    echo ""
+    print_success "counterweight user created on all VPSs"
+    print_success "SSH key authentication configured"
+    print_success "UFW firewall active and configured"
+    print_success "fail2ban protecting against brute force attacks"
+    print_success "auditd logging security events"
+    echo ""
+    
+    print_warning "Important Security Changes:"
+    echo "  • Root SSH login is now disabled (by design)"
+    echo "  • Always use 'counterweight' user for SSH access"
+    echo "  • Firewall is active - only SSH allowed by default"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Test SSH access: ssh -i ~/.ssh/counterganzua counterweight@<host>"
+    echo "  2. (Optional) Set up Nodito: ./scripts/setup_layer_1b_nodito.sh"
+    echo "  3. Proceed to Layer 2: ./scripts/setup_layer_2.sh"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "🔧 Layer 1A: VPS Basic Setup"
+    
+    echo "This script will configure users, SSH, firewall, and fail2ban on VPS machines."
+    echo ""
+    print_info "Targets: vipy, watchtower, spacey"
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 1A setup?"; then
+        echo "Setup cancelled."
+        exit 0
+    fi
+    
+    check_layer_0_complete
+    check_vps_configured
+    check_ssh_connectivity
+    
+    # VPS Setup
+    local setup_failed=false
+    setup_vps_users_and_access || setup_failed=true
+    setup_vps_firewall_and_fail2ban || setup_failed=true
+    
+    verify_layer_1a
+    
+    if [ "$setup_failed" = true ]; then
+        print_warning "Some steps failed - please review errors above"
+    fi
+    
+    print_summary
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_1b_nodito.sh

@@ -0,0 +1,411 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 1B: Nodito (Proxmox) Setup
+# 
+# This script configures the Nodito Proxmox server.
+# Runs independently - can be executed without VPS setup.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+check_layer_0_complete() {
+    print_header "Verifying Layer 0 Prerequisites"
+    
+    local errors=0
+    
+    # Check if venv exists
+    if [ ! -d "$PROJECT_ROOT/venv" ]; then
+        print_error "Python venv not found. Run Layer 0 first."
+        ((errors++))
+    else
+        print_success "Python venv exists"
+    fi
+    
+    # Check if we're in a venv
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+    
+    # Check if Ansible is installed
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found: $(ansible --version | head -n1)"
+    fi
+    
+    # Check if inventory.ini exists
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+    
+    if [ $errors -gt 0 ]; then
+        print_error "Layer 0 is not complete. Please run ./scripts/setup_layer_0.sh first"
+        exit 1
+    fi
+    
+    print_success "Layer 0 prerequisites verified"
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+if target in data:
+    print(' '.join(data[target].get('hosts', [])))
+else:
+    hostvars = data.get('_meta', {}).get('hostvars', {})
+    if target in hostvars:
+        print(target)
+PY
+}
+
+check_nodito_configured() {
+    print_header "Checking Nodito Configuration"
+    
+    local nodito_hosts=$(get_hosts_from_inventory "nodito_host")
+    
+    if [ -z "$nodito_hosts" ]; then
+        print_error "No nodito host configured in inventory.ini"
+        print_info "Add the nodito host to the [nodito_host] group in inventory.ini to proceed"
+        exit 1
+    fi
+    
+    print_success "Nodito configured: $nodito_hosts"
+    echo ""
+}
+
+###############################################################################
+# Nodito Setup Functions
+###############################################################################
+
+setup_nodito_bootstrap() {
+    print_header "Bootstrapping Nodito (Proxmox Server)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Set up SSH key access for root"
+    echo "  • Create the counterweight user with SSH keys"
+    echo "  • Update and secure the system"
+    echo "  • Disable root login and password authentication"
+    echo ""
+    print_info "Running: ansible-playbook -i inventory.ini infra/nodito/30_proxmox_bootstrap_playbook.yml"
+    print_warning "You will be prompted for the root password"
+    echo ""
+    
+    if ! confirm_action "Proceed with nodito bootstrap?"; then
+        print_warning "Skipped nodito bootstrap"
+        return 1
+    fi
+    
+    # Run with root user and ask for password
+    if ansible-playbook -i inventory.ini infra/nodito/30_proxmox_bootstrap_playbook.yml -e 'ansible_user=root' --ask-pass; then
+        print_success "Nodito bootstrap complete"
+        return 0
+    else
+        print_error "Nodito bootstrap failed"
+        return 1
+    fi
+}
+
+setup_nodito_community_repos() {
+    print_header "Switching Nodito to Community Repositories"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Remove enterprise repository files"
+    echo "  • Add community repository files"
+    echo "  • Disable subscription nag messages"
+    echo "  • Update Proxmox packages"
+    echo ""
+    print_info "Running: ansible-playbook -i inventory.ini infra/nodito/31_proxmox_community_repos_playbook.yml"
+    echo ""
+    
+    if ! confirm_action "Proceed with community repos setup?"; then
+        print_warning "Skipped community repos setup"
+        return 1
+    fi
+    
+    if ansible-playbook -i inventory.ini infra/nodito/31_proxmox_community_repos_playbook.yml; then
+        print_success "Community repositories configured"
+        print_warning "Clear browser cache before using Proxmox web UI (Ctrl+Shift+R)"
+        return 0
+    else
+        print_error "Community repos setup failed"
+        return 1
+    fi
+}
+
+setup_nodito_zfs() {
+    print_header "Setting Up ZFS Storage Pool on Nodito (Optional)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_warning "⚠️  ZFS setup will DESTROY ALL DATA on the specified disks!"
+    echo ""
+    print_info "Before proceeding, you must:"
+    echo "  1. SSH into nodito: ssh root@<nodito-ip>"
+    echo "  2. List disks: ls -la /dev/disk/by-id/ | grep -E '(ata-|scsi-|nvme-)'"
+    echo "  3. Identify the two disk IDs you want to use for RAID 1"
+    echo "  4. Edit ansible/infra/nodito/nodito_vars.yml"
+    echo "  5. Set zfs_disk_1 and zfs_disk_2 to your disk IDs"
+    echo ""
+    print_info "Example nodito_vars.yml content:"
+    echo '  zfs_disk_1: "/dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1234567"'
+    echo '  zfs_disk_2: "/dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K7654321"'
+    echo ""
+    
+    if [ ! -f "$ANSIBLE_DIR/infra/nodito/nodito_vars.yml" ]; then
+        print_warning "nodito_vars.yml not found"
+        if confirm_action "Create nodito_vars.yml template?"; then
+            cat > "$ANSIBLE_DIR/infra/nodito/nodito_vars.yml" << 'EOF'
+# Nodito Variables
+# Configure these before running ZFS setup
+
+# ZFS Storage Pool Configuration
+# Uncomment and configure these lines after identifying your disk IDs:
+# zfs_disk_1: "/dev/disk/by-id/ata-YOUR-DISK-1-ID-HERE"
+# zfs_disk_2: "/dev/disk/by-id/ata-YOUR-DISK-2-ID-HERE"
+# zfs_pool_name: "proxmox-storage"
+
+# CPU Temperature Monitoring
+monitoring_script_dir: /opt/cpu-temp-monitor
+monitoring_script_path: "{{ monitoring_script_dir }}/cpu_temp_monitor.sh"
+log_file: "{{ monitoring_script_dir }}/cpu_temp_monitor.log"
+temp_threshold_celsius: 80
+EOF
+            print_success "Created nodito_vars.yml template"
+            print_info "Edit this file and configure ZFS disks, then re-run this script"
+        fi
+        return 1
+    fi
+    
+    # Check if ZFS disks are configured
+    if ! grep -q "^zfs_disk_1:" "$ANSIBLE_DIR/infra/nodito/nodito_vars.yml" 2>/dev/null; then
+        print_info "ZFS disks not configured in nodito_vars.yml"
+        print_info "Edit ansible/infra/nodito/nodito_vars.yml to configure disk IDs"
+        if ! confirm_action "Skip ZFS setup for now?"; then
+            print_info "Please configure ZFS disks first"
+            return 1
+        fi
+        print_warning "Skipped ZFS setup"
+        return 1
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini infra/nodito/32_zfs_pool_setup_playbook.yml"
+    echo ""
+    
+    if ! confirm_action "⚠️  Proceed with ZFS setup? (THIS WILL DESTROY DATA ON CONFIGURED DISKS)"; then
+        print_warning "Skipped ZFS setup"
+        return 1
+    fi
+    
+    if ansible-playbook -i inventory.ini infra/nodito/32_zfs_pool_setup_playbook.yml; then
+        print_success "ZFS storage pool configured"
+        return 0
+    else
+        print_error "ZFS setup failed"
+        return 1
+    fi
+}
+
+setup_nodito_cloud_template() {
+    print_header "Creating Debian Cloud Template on Nodito (Optional)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Download Debian cloud image"
+    echo "  • Create a VM template (ID 9000)"
+    echo "  • Configure cloud-init for easy VM creation"
+    echo ""
+    print_info "Running: ansible-playbook -i inventory.ini infra/nodito/33_proxmox_debian_cloud_template.yml"
+    echo ""
+    
+    if ! confirm_action "Proceed with cloud template creation?"; then
+        print_warning "Skipped cloud template creation"
+        return 1
+    fi
+    
+    if ansible-playbook -i inventory.ini infra/nodito/33_proxmox_debian_cloud_template.yml; then
+        print_success "Debian cloud template created (VM ID 9000)"
+        return 0
+    else
+        print_error "Cloud template creation failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+verify_layer_1b() {
+    print_header "Verifying Layer 1B Completion"
+    
+    cd "$ANSIBLE_DIR"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    local nodito_hosts=$(get_hosts_from_inventory "nodito")
+    
+    print_info "Testing SSH as counterweight user..."
+    echo ""
+    
+    for host in $nodito_hosts; do
+        if timeout 10 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "echo 'SSH OK'" &>/dev/null; then
+            print_success "SSH to $host as counterweight: OK"
+        else
+            print_error "Cannot SSH to $host as counterweight"
+            print_info "Test manually: ssh -i $ssh_key counterweight@$host"
+        fi
+    done
+    
+    echo ""
+}
+
+###############################################################################
+# Summary Functions
+###############################################################################
+
+print_summary() {
+    print_header "Layer 1B: Nodito Setup Complete! 🎉"
+    
+    echo "Summary of what was configured:"
+    echo ""
+    print_success "Nodito bootstrapped with SSH keys"
+    print_success "counterweight user created"
+    print_success "Community repositories configured"
+    print_success "Root login and password auth disabled"
+    
+    if grep -q "^zfs_disk_1:" "$ANSIBLE_DIR/infra/nodito/nodito_vars.yml" 2>/dev/null; then
+        print_success "ZFS storage pool configured (if you ran it)"
+    fi
+    echo ""
+    
+    print_warning "Important Security Changes:"
+    echo "  • Root SSH login is now disabled"
+    echo "  • Always use 'counterweight' user for SSH access"
+    echo "  • Password authentication is disabled"
+    echo ""
+    
+    print_info "Proxmox Web UI:"
+    local nodito_hosts=$(get_hosts_from_inventory "nodito")
+    echo "  • Access at: https://$nodito_hosts:8006"
+    echo "  • Clear browser cache (Ctrl+Shift+R) to avoid UI issues"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Test SSH: ssh -i ~/.ssh/counterganzua counterweight@<nodito-ip>"
+    echo "  2. Access Proxmox web UI and verify community repos"
+    echo "  3. Create VMs on Proxmox (if needed)"
+    echo "  4. Proceed to Layer 2: ./scripts/setup_layer_2.sh"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "🖥️  Layer 1B: Nodito (Proxmox) Setup"
+    
+    echo "This script will configure your Nodito Proxmox server."
+    echo ""
+    print_info "Target: nodito (Proxmox server)"
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 1B setup?"; then
+        echo "Setup cancelled."
+        exit 0
+    fi
+    
+    check_layer_0_complete
+    check_nodito_configured
+    
+    # Nodito Setup
+    local setup_failed=false
+    setup_nodito_bootstrap || setup_failed=true
+    setup_nodito_community_repos || setup_failed=true
+    setup_nodito_zfs || setup_failed=true
+    setup_nodito_cloud_template || setup_failed=true
+    
+    verify_layer_1b
+    
+    if [ "$setup_failed" = true ]; then
+        print_warning "Some optional steps were skipped - this is normal"
+    fi
+    
+    print_summary
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_2.sh

@@ -0,0 +1,407 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 2: General Infrastructure Tools
+# 
+# This script installs rsync and docker on the machines that need them.
+# Must be run after Layer 1A (VPS) or Layer 1B (Nodito) is complete.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+check_layer_0_complete() {
+    print_header "Verifying Layer 0 Prerequisites"
+    
+    local errors=0
+    
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+    
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found"
+    fi
+    
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+    
+    if [ $errors -gt 0 ]; then
+        print_error "Layer 0 is not complete"
+        exit 1
+    fi
+    
+    print_success "Layer 0 prerequisites verified"
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+if target in data:
+    print(' '.join(data[target].get('hosts', [])))
+else:
+    hostvars = data.get('_meta', {}).get('hostvars', {})
+    if target in hostvars:
+        print(target)
+PY
+}
+
+check_ssh_connectivity() {
+    print_header "Testing SSH Connectivity"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    local all_good=true
+    
+    for group in vipy watchtower spacey nodito; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            for host in $hosts; do
+                print_info "Testing SSH to $host as counterweight..."
+                if timeout 10 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "echo 'SSH OK'" &>/dev/null; then
+                    print_success "SSH to $host: OK"
+                else
+                    print_error "Cannot SSH to $host as counterweight"
+                    print_warning "Make sure Layer 1A or 1B is complete for this host"
+                    all_good=false
+                fi
+            done
+        fi
+    done
+    
+    if [ "$all_good" = false ]; then
+        echo ""
+        print_error "SSH connectivity test failed"
+        print_info "Ensure Layer 1A (VPS) or Layer 1B (Nodito) is complete"
+        echo ""
+        if ! confirm_action "Continue anyway?"; then
+            exit 1
+        fi
+    fi
+    
+    echo ""
+    print_success "SSH connectivity verified"
+}
+
+###############################################################################
+# rsync Installation
+###############################################################################
+
+install_rsync() {
+    print_header "Installing rsync"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "rsync is needed for backup operations"
+    print_info "Recommended hosts: vipy, watchtower, lapy"
+    echo ""
+    
+    # Show available hosts
+    echo "Available hosts in inventory:"
+    for group in vipy watchtower spacey nodito lapy; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            echo "  [$group]: $hosts"
+        fi
+    done
+    echo ""
+    
+    print_info "Installation options:"
+    echo "  1. Install on recommended hosts (vipy, watchtower, lapy)"
+    echo "  2. Install on all hosts"
+    echo "  3. Custom selection (specify groups)"
+    echo "  4. Skip rsync installation"
+    echo ""
+    
+    echo -e -n "${BLUE}Choose option${NC} [1-4]: "
+    read option
+    
+    local limit_hosts=""
+    case "$option" in
+        1)
+            limit_hosts="vipy,watchtower,lapy"
+            print_info "Installing rsync on: vipy, watchtower, lapy"
+            ;;
+        2)
+            limit_hosts="all"
+            print_info "Installing rsync on: all hosts"
+            ;;
+        3)
+            echo -e -n "${BLUE}Enter groups (comma-separated, e.g., vipy,watchtower,nodito)${NC}: "
+            read limit_hosts
+            print_info "Installing rsync on: $limit_hosts"
+            ;;
+        4)
+            print_warning "Skipping rsync installation"
+            return 1
+            ;;
+        *)
+            print_error "Invalid option"
+            return 1
+            ;;
+    esac
+    
+    echo ""
+    if ! confirm_action "Proceed with rsync installation?"; then
+        print_warning "Skipped rsync installation"
+        return 1
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini infra/900_install_rsync.yml --limit $limit_hosts"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini infra/900_install_rsync.yml --limit "$limit_hosts"; then
+        print_success "rsync installation complete"
+        return 0
+    else
+        print_error "rsync installation failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Docker Installation
+###############################################################################
+
+install_docker() {
+    print_header "Installing Docker and Docker Compose"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "Docker is needed for containerized services"
+    print_info "Recommended hosts: vipy, watchtower"
+    echo ""
+    
+    # Show available hosts (exclude lapy - docker on laptop is optional)
+    echo "Available hosts in inventory:"
+    for group in vipy watchtower spacey nodito; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            echo "  [$group]: $hosts"
+        fi
+    done
+    echo ""
+    
+    print_info "Installation options:"
+    echo "  1. Install on recommended hosts (vipy, watchtower)"
+    echo "  2. Install on all hosts"
+    echo "  3. Custom selection (specify groups)"
+    echo "  4. Skip docker installation"
+    echo ""
+    
+    echo -e -n "${BLUE}Choose option${NC} [1-4]: "
+    read option
+    
+    local limit_hosts=""
+    case "$option" in
+        1)
+            limit_hosts="vipy,watchtower"
+            print_info "Installing Docker on: vipy, watchtower"
+            ;;
+        2)
+            limit_hosts="all"
+            print_info "Installing Docker on: all hosts"
+            ;;
+        3)
+            echo -e -n "${BLUE}Enter groups (comma-separated, e.g., vipy,watchtower,nodito)${NC}: "
+            read limit_hosts
+            print_info "Installing Docker on: $limit_hosts"
+            ;;
+        4)
+            print_warning "Skipping Docker installation"
+            return 1
+            ;;
+        *)
+            print_error "Invalid option"
+            return 1
+            ;;
+    esac
+    
+    echo ""
+    if ! confirm_action "Proceed with Docker installation?"; then
+        print_warning "Skipped Docker installation"
+        return 1
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini infra/910_docker_playbook.yml --limit $limit_hosts"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini infra/910_docker_playbook.yml --limit "$limit_hosts"; then
+        print_success "Docker installation complete"
+        print_warning "You may need to log out and back in for docker group to take effect"
+        return 0
+    else
+        print_error "Docker installation failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+verify_installations() {
+    print_header "Verifying Installations"
+    
+    cd "$ANSIBLE_DIR"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    echo "Checking installed tools on hosts..."
+    echo ""
+    
+    # Check all remote hosts
+    for group in vipy watchtower spacey nodito; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            for host in $hosts; do
+                print_info "Checking $host..."
+                
+                # Check rsync
+                if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "command -v rsync" &>/dev/null; then
+                    print_success "$host: rsync installed"
+                else
+                    print_warning "$host: rsync not found (may not be needed)"
+                fi
+                
+                # Check docker
+                if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "command -v docker" &>/dev/null; then
+                    print_success "$host: docker installed"
+                    
+                    # Check docker service
+                    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "sudo systemctl is-active docker" &>/dev/null; then
+                        print_success "$host: docker service running"
+                    else
+                        print_warning "$host: docker service not running"
+                    fi
+                else
+                    print_warning "$host: docker not found (may not be needed)"
+                fi
+                
+                echo ""
+            done
+        fi
+    done
+}
+
+###############################################################################
+# Summary Functions
+###############################################################################
+
+print_summary() {
+    print_header "Layer 2 Setup Complete! 🎉"
+    
+    echo "Summary:"
+    echo ""
+    print_success "Infrastructure tools installed on specified hosts"
+    echo ""
+    
+    print_info "What was installed:"
+    echo "  • rsync - for backup operations"
+    echo "  • docker + docker compose - for containerized services"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Proceed to Layer 3: ./scripts/setup_layer_3_caddy.sh"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "🔧 Layer 2: General Infrastructure Tools"
+    
+    echo "This script will install rsync and docker on your infrastructure."
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 2 setup?"; then
+        echo "Setup cancelled."
+        exit 0
+    fi
+    
+    check_layer_0_complete
+    check_ssh_connectivity
+    
+    # Install tools
+    install_rsync
+    echo ""
+    install_docker
+    
+    verify_installations
+    print_summary
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_3_caddy.sh

@@ -0,0 +1,355 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 3: Reverse Proxy (Caddy)
+# 
+# This script deploys Caddy reverse proxy on VPS machines.
+# Must be run after Layer 1A (VPS setup) is complete.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+check_layer_0_complete() {
+    print_header "Verifying Layer 0 Prerequisites"
+    
+    local errors=0
+    
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+    
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found"
+    fi
+    
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+    
+    if [ $errors -gt 0 ]; then
+        print_error "Layer 0 is not complete"
+        exit 1
+    fi
+    
+    print_success "Layer 0 prerequisites verified"
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+if target in data:
+    print(' '.join(data[target].get('hosts', [])))
+else:
+    hostvars = data.get('_meta', {}).get('hostvars', {})
+    if target in hostvars:
+        print(target)
+PY
+}
+
+check_target_hosts() {
+    print_header "Checking Target Hosts"
+    
+    local has_hosts=false
+    
+    print_info "Caddy will be deployed to these hosts:"
+    echo ""
+    
+    for group in vipy watchtower spacey; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            echo "  [$group]: $hosts"
+            has_hosts=true
+        else
+            print_warning "[$group]: not configured (skipping)"
+        fi
+    done
+    
+    echo ""
+    
+    if [ "$has_hosts" = false ]; then
+        print_error "No target hosts configured for Caddy"
+        print_info "Caddy needs vipy, watchtower, or spacey in inventory.ini"
+        exit 1
+    fi
+    
+    print_success "Target hosts verified"
+}
+
+check_ssh_connectivity() {
+    print_header "Testing SSH Connectivity"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    local all_good=true
+    
+    for group in vipy watchtower spacey; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            for host in $hosts; do
+                print_info "Testing SSH to $host as counterweight..."
+                if timeout 10 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "echo 'SSH OK'" &>/dev/null; then
+                    print_success "SSH to $host: OK"
+                else
+                    print_error "Cannot SSH to $host as counterweight"
+                    print_warning "Make sure Layer 1A is complete for this host"
+                    all_good=false
+                fi
+            done
+        fi
+    done
+    
+    if [ "$all_good" = false ]; then
+        echo ""
+        print_error "SSH connectivity test failed"
+        print_info "Ensure Layer 1A (VPS setup) is complete"
+        echo ""
+        if ! confirm_action "Continue anyway?"; then
+            exit 1
+        fi
+    fi
+    
+    echo ""
+    print_success "SSH connectivity verified"
+}
+
+###############################################################################
+# Caddy Deployment
+###############################################################################
+
+deploy_caddy() {
+    print_header "Deploying Caddy"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Install Caddy from official repositories"
+    echo "  • Configure Caddy service"
+    echo "  • Open firewall ports 80/443"
+    echo "  • Create sites-enabled directory structure"
+    echo "  • Enable automatic HTTPS with Let's Encrypt"
+    echo ""
+    
+    print_info "Target hosts: vipy, watchtower, spacey (if configured)"
+    echo ""
+    
+    print_warning "Important:"
+    echo "  • Caddy will start with empty configuration"
+    echo "  • Services will add their own config files in later layers"
+    echo "  • Ports 80/443 must be available on the VPSs"
+    echo ""
+    
+    if ! confirm_action "Proceed with Caddy deployment?"; then
+        print_warning "Skipped Caddy deployment"
+        return 1
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/caddy_playbook.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/caddy_playbook.yml; then
+        print_success "Caddy deployment complete"
+        return 0
+    else
+        print_error "Caddy deployment failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+verify_caddy() {
+    print_header "Verifying Caddy Installation"
+    
+    cd "$ANSIBLE_DIR"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    echo "Checking Caddy on each host..."
+    echo ""
+    
+    for group in vipy watchtower spacey; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            for host in $hosts; do
+                print_info "Checking $host..."
+                
+                # Check if caddy is installed
+                if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "command -v caddy" &>/dev/null; then
+                    print_success "$host: Caddy installed"
+                else
+                    print_error "$host: Caddy not found"
+                    continue
+                fi
+                
+                # Check if caddy service is running
+                if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "sudo systemctl is-active caddy" &>/dev/null; then
+                    print_success "$host: Caddy service running"
+                else
+                    print_error "$host: Caddy service not running"
+                fi
+                
+                # Check if sites-enabled directory exists
+                if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "test -d /etc/caddy/sites-enabled" &>/dev/null; then
+                    print_success "$host: sites-enabled directory exists"
+                else
+                    print_warning "$host: sites-enabled directory not found"
+                fi
+                
+                # Check if ports 80/443 are open
+                if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$host "sudo ufw status | grep -E '80|443'" &>/dev/null; then
+                    print_success "$host: Firewall ports 80/443 open"
+                else
+                    print_warning "$host: Could not verify firewall ports"
+                fi
+                
+                echo ""
+            done
+        fi
+    done
+}
+
+###############################################################################
+# Summary Functions
+###############################################################################
+
+print_summary() {
+    print_header "Layer 3 Setup Complete! 🎉"
+    
+    echo "Summary of what was configured:"
+    echo ""
+    print_success "Caddy installed on VPS hosts"
+    print_success "Caddy service running"
+    print_success "Firewall ports 80/443 opened"
+    print_success "Sites-enabled directory structure created"
+    echo ""
+    
+    print_info "What Caddy provides:"
+    echo "  • Automatic HTTPS with Let's Encrypt"
+    echo "  • Reverse proxy for all web services"
+    echo "  • HTTP/2 support"
+    echo "  • Simple per-service configuration"
+    echo ""
+    
+    print_info "How services use Caddy:"
+    echo "  • Each service adds a config file to /etc/caddy/sites-enabled/"
+    echo "  • Main Caddyfile imports all configs"
+    echo "  • Caddy automatically manages SSL certificates"
+    echo ""
+    
+    print_warning "Important Notes:"
+    echo "  • Caddy is currently running with default/empty config"
+    echo "  • Services deployed in later layers will add their configs"
+    echo "  • DNS must point to your VPS IPs for SSL to work"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Verify Caddy is accessible (optional): curl http://<vps-ip>"
+    echo "  2. Proceed to Layer 4: ./scripts/setup_layer_4_monitoring.sh"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "🌐 Layer 3: Reverse Proxy (Caddy)"
+    
+    echo "This script will deploy Caddy reverse proxy on your VPS machines."
+    echo ""
+    print_info "Targets: vipy, watchtower, spacey"
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 3 setup?"; then
+        echo "Setup cancelled."
+        exit 0
+    fi
+    
+    check_layer_0_complete
+    check_target_hosts
+    check_ssh_connectivity
+    
+    # Deploy Caddy
+    if deploy_caddy; then
+        verify_caddy
+        print_summary
+    else
+        print_error "Caddy deployment failed"
+        exit 1
+    fi
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_4_monitoring.sh

@@ -0,0 +1,806 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 4: Core Monitoring & Notifications
+# 
+# This script deploys ntfy and Uptime Kuma on watchtower.
+# Must be run after Layers 1A, 2, and 3 are complete.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+if target in data:
+    print(' '.join(data[target].get('hosts', [])))
+else:
+    hostvars = data.get('_meta', {}).get('hostvars', {})
+    if target in hostvars:
+        print(target)
+PY
+}
+
+get_host_ip() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+hostvars = data.get('_meta', {}).get('hostvars', {})
+if target in hostvars:
+    print(hostvars[target].get('ansible_host', target))
+else:
+    hosts = data.get(target, {}).get('hosts', [])
+    if hosts:
+        first = hosts[0]
+        hv = hostvars.get(first, {})
+        print(hv.get('ansible_host', first))
+PY
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+check_prerequisites() {
+    print_header "Verifying Prerequisites"
+    
+    local errors=0
+    
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+    
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found"
+    fi
+    
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+    
+    # Check if watchtower is configured
+    if [ -z "$(get_hosts_from_inventory "watchtower")" ]; then
+        print_error "watchtower not configured in inventory.ini"
+        print_info "Layer 4 requires watchtower VPS"
+        ((errors++))
+    else
+        print_success "watchtower configured in inventory"
+    fi
+    
+    if [ $errors -gt 0 ]; then
+        print_error "Prerequisites not met"
+        exit 1
+    fi
+    
+    print_success "Prerequisites verified"
+}
+
+check_vars_files() {
+    print_header "Checking Configuration Files"
+    
+    # Check services_config.yml
+    if [ ! -f "$ANSIBLE_DIR/services_config.yml" ]; then
+        print_error "services_config.yml not found"
+        print_info "This file should have been created in Layer 0"
+        exit 1
+    fi
+    
+    print_success "services_config.yml exists"
+    
+    # Show configured subdomains
+    local ntfy_sub=$(grep "^  ntfy:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "ntfy")
+    local uptime_sub=$(grep "^  uptime_kuma:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "uptime")
+    
+    print_info "Configured subdomains:"
+    echo "  • ntfy: $ntfy_sub"
+    echo "  • uptime_kuma: $uptime_sub"
+    echo ""
+}
+
+check_dns_configuration() {
+    print_header "Validating DNS Configuration"
+    
+    cd "$ANSIBLE_DIR"
+    
+    # Get watchtower IP
+    local watchtower_ip=$(get_host_ip "watchtower")
+    
+    if [ -z "$watchtower_ip" ]; then
+        print_error "Could not determine watchtower IP from inventory"
+        return 1
+    fi
+    
+    print_info "Watchtower IP: $watchtower_ip"
+    echo ""
+    
+    # Get domain from infra_vars.yml
+    local root_domain=$(grep "^root_domain:" "$ANSIBLE_DIR/infra_vars.yml" | awk '{print $2}' 2>/dev/null)
+    
+    if [ -z "$root_domain" ]; then
+        print_error "Could not determine root_domain from infra_vars.yml"
+        return 1
+    fi
+    
+    # Get subdomains from centralized config
+    local ntfy_subdomain="ntfy"
+    local uptime_subdomain="uptime"
+    
+    if [ -f "$ANSIBLE_DIR/services_config.yml" ]; then
+        ntfy_subdomain=$(grep "^  ntfy:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "ntfy")
+        uptime_subdomain=$(grep "^  uptime_kuma:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "uptime")
+    fi
+    
+    local ntfy_fqdn="${ntfy_subdomain}.${root_domain}"
+    local uptime_fqdn="${uptime_subdomain}.${root_domain}"
+    
+    print_info "Checking DNS records..."
+    echo ""
+    
+    local dns_ok=true
+    
+    # Check ntfy DNS
+    print_info "Checking $ntfy_fqdn..."
+    if command -v dig &> /dev/null; then
+        local ntfy_resolved=$(dig +short "$ntfy_fqdn" | head -n1)
+        if [ "$ntfy_resolved" = "$watchtower_ip" ]; then
+            print_success "$ntfy_fqdn → $ntfy_resolved ✓"
+        elif [ -n "$ntfy_resolved" ]; then
+            print_error "$ntfy_fqdn → $ntfy_resolved (expected $watchtower_ip)"
+            dns_ok=false
+        else
+            print_error "$ntfy_fqdn does not resolve"
+            dns_ok=false
+        fi
+    else
+        print_warning "dig command not found, skipping DNS validation"
+        print_info "Install dnsutils/bind-tools to enable DNS validation"
+        return 1
+    fi
+    
+    # Check Uptime Kuma DNS
+    print_info "Checking $uptime_fqdn..."
+    if command -v dig &> /dev/null; then
+        local uptime_resolved=$(dig +short "$uptime_fqdn" | head -n1)
+        if [ "$uptime_resolved" = "$watchtower_ip" ]; then
+            print_success "$uptime_fqdn → $uptime_resolved ✓"
+        elif [ -n "$uptime_resolved" ]; then
+            print_error "$uptime_fqdn → $uptime_resolved (expected $watchtower_ip)"
+            dns_ok=false
+        else
+            print_error "$uptime_fqdn does not resolve"
+            dns_ok=false
+        fi
+    fi
+    
+    echo ""
+    
+    if [ "$dns_ok" = false ]; then
+        print_error "DNS validation failed"
+        print_info "Please configure DNS records:"
+        echo "  • $ntfy_fqdn → $watchtower_ip"
+        echo "  • $uptime_fqdn → $watchtower_ip"
+        echo ""
+        print_warning "DNS changes can take time to propagate (up to 24-48 hours)"
+        echo ""
+        if ! confirm_action "Continue anyway? (SSL certificates will fail without proper DNS)"; then
+            exit 1
+        fi
+    else
+        print_success "DNS validation passed"
+    fi
+}
+
+###############################################################################
+# ntfy Deployment
+###############################################################################
+
+deploy_ntfy() {
+    print_header "Deploying ntfy (Notification Service)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "ntfy requires admin credentials for authentication"
+    echo ""
+    
+    # Check if env vars are set
+    if [ -z "$NTFY_USER" ] || [ -z "$NTFY_PASSWORD" ]; then
+        print_warning "NTFY_USER and NTFY_PASSWORD environment variables not set"
+        echo ""
+        print_info "Please enter credentials for ntfy admin user:"
+        echo ""
+        
+        echo -e -n "${BLUE}ntfy admin username${NC} [admin]: "
+        read ntfy_user
+        ntfy_user="${ntfy_user:-admin}"
+        
+        echo -e -n "${BLUE}ntfy admin password${NC}: "
+        read -s ntfy_password
+        echo ""
+        
+        if [ -z "$ntfy_password" ]; then
+            print_error "Password cannot be empty"
+            return 1
+        fi
+        
+        export NTFY_USER="$ntfy_user"
+        export NTFY_PASSWORD="$ntfy_password"
+    else
+        print_success "Using NTFY_USER and NTFY_PASSWORD from environment"
+    fi
+    
+    echo ""
+    print_info "This will:"
+    echo "  • Install ntfy from official repositories"
+    echo "  • Configure ntfy with authentication (deny-all by default)"
+    echo "  • Create admin user: $NTFY_USER"
+    echo "  • Set up Caddy reverse proxy"
+    echo ""
+    
+    if ! confirm_action "Proceed with ntfy deployment?"; then
+        print_warning "Skipped ntfy deployment"
+        return 1
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/ntfy/deploy_ntfy_playbook.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/ntfy/deploy_ntfy_playbook.yml; then
+        print_success "ntfy deployment complete"
+        echo ""
+        print_info "ntfy is now available at your configured subdomain"
+        print_info "Admin user: $NTFY_USER"
+        return 0
+    else
+        print_error "ntfy deployment failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Uptime Kuma Deployment
+###############################################################################
+
+deploy_uptime_kuma() {
+    print_header "Deploying Uptime Kuma (Monitoring Platform)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Deploy Uptime Kuma via Docker"
+    echo "  • Configure Caddy reverse proxy"
+    echo "  • Set up data persistence"
+    echo ""
+    
+    if ! confirm_action "Proceed with Uptime Kuma deployment?"; then
+        print_warning "Skipped Uptime Kuma deployment"
+        return 1
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/uptime_kuma/deploy_uptime_kuma_playbook.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/uptime_kuma/deploy_uptime_kuma_playbook.yml; then
+        print_success "Uptime Kuma deployment complete"
+        echo ""
+        print_warning "IMPORTANT: First-time setup required"
+        echo "  1. Access Uptime Kuma at your configured subdomain"
+        echo "  2. Create admin user on first visit"
+        echo "  3. Update ansible/infra_secrets.yml with credentials"
+        return 0
+    else
+        print_error "Uptime Kuma deployment failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Backup Configuration
+###############################################################################
+
+setup_uptime_kuma_backup() {
+    print_header "Setting Up Uptime Kuma Backup (Optional)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will set up automated backups to lapy"
+    echo ""
+    
+    if ! confirm_action "Set up Uptime Kuma backup to lapy?"; then
+        print_warning "Skipped backup setup"
+        return 0
+    fi
+    
+    # Check if rsync is available
+    print_info "Verifying rsync is installed on watchtower and lapy..."
+    if ! ansible watchtower -i inventory.ini -m shell -a "command -v rsync" &>/dev/null; then
+        print_error "rsync not found on watchtower"
+        print_info "Run Layer 2 to install rsync"
+        print_warning "Backup setup skipped - rsync not available"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml; then
+        print_success "Uptime Kuma backup configured"
+        print_info "Backups will run periodically via cron"
+        return 0
+    else
+        print_error "Backup setup failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Post-Deployment Configuration
+###############################################################################
+
+setup_ntfy_notification() {
+    print_header "Setting Up ntfy Notification in Uptime Kuma (Optional)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will automatically configure ntfy as a notification method in Uptime Kuma"
+    print_warning "Prerequisites:"
+    echo "  • Uptime Kuma admin account must be created first"
+    echo "  • infra_secrets.yml must have Uptime Kuma credentials"
+    echo ""
+    
+    if ! confirm_action "Set up ntfy notification in Uptime Kuma?"; then
+        print_warning "Skipped ntfy notification setup"
+        print_info "You can set this up manually or run this script again later"
+        return 0
+    fi
+    
+    # Check if infra_secrets.yml has Uptime Kuma credentials
+    if ! grep -q "uptime_kuma_username:" "$ANSIBLE_DIR/infra_secrets.yml" 2>/dev/null || \
+       ! grep -q "uptime_kuma_password:" "$ANSIBLE_DIR/infra_secrets.yml" 2>/dev/null; then
+        print_error "Uptime Kuma credentials not found in infra_secrets.yml"
+        print_info "Please complete Step 1 and 2 of post-deployment steps first:"
+        echo "  1. Create admin user in Uptime Kuma web UI"
+        echo "  2. Add credentials to ansible/infra_secrets.yml"
+        print_warning "Skipped - you can run this script again after completing those steps"
+        return 0
+    fi
+    
+    # Check credentials are not empty
+    local uk_user=$(grep "^uptime_kuma_username:" "$ANSIBLE_DIR/infra_secrets.yml" | awk '{print $2}' | tr -d '"' | tr -d "'")
+    local uk_pass=$(grep "^uptime_kuma_password:" "$ANSIBLE_DIR/infra_secrets.yml" | awk '{print $2}' | tr -d '"' | tr -d "'")
+    
+    if [ -z "$uk_user" ] || [ -z "$uk_pass" ]; then
+        print_error "Uptime Kuma credentials are empty in infra_secrets.yml"
+        print_info "Please update ansible/infra_secrets.yml with your credentials"
+        return 0
+    fi
+    
+    print_success "Found Uptime Kuma credentials in infra_secrets.yml"
+    
+    print_info "Running playbook to configure ntfy notification..."
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/ntfy/setup_ntfy_uptime_kuma_notification.yml; then
+        print_success "ntfy notification configured in Uptime Kuma"
+        print_info "You can now use ntfy for all your monitors!"
+        return 0
+    else
+        print_error "Failed to configure ntfy notification"
+        print_info "You can set this up manually or run the playbook again later:"
+        echo "  ansible-playbook -i inventory.ini services/ntfy/setup_ntfy_uptime_kuma_notification.yml"
+        return 0
+    fi
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+verify_deployments() {
+    print_header "Verifying Deployments"
+    
+    cd "$ANSIBLE_DIR"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    local watchtower_host
+    watchtower_host=$(get_hosts_from_inventory "watchtower")
+    
+    if [ -z "$watchtower_host" ]; then
+        print_error "Could not determine watchtower host"
+        return
+    fi
+    
+    print_info "Checking services on watchtower ($watchtower_host)..."
+    echo ""
+    
+    # Check ntfy
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$watchtower_host "systemctl is-active ntfy" &>/dev/null; then
+        print_success "ntfy service running"
+    else
+        print_warning "ntfy service not running or not installed"
+    fi
+    
+    # Check Uptime Kuma docker container
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$watchtower_host "docker ps | grep uptime-kuma" &>/dev/null; then
+        print_success "Uptime Kuma container running"
+    else
+        print_warning "Uptime Kuma container not running"
+    fi
+    
+    # Check Caddy configs
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$watchtower_host "test -f /etc/caddy/sites-enabled/ntfy.conf" &>/dev/null; then
+        print_success "ntfy Caddy config exists"
+    else
+        print_warning "ntfy Caddy config not found"
+    fi
+    
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$watchtower_host "test -f /etc/caddy/sites-enabled/uptime-kuma.conf" &>/dev/null; then
+        print_success "Uptime Kuma Caddy config exists"
+    else
+        print_warning "Uptime Kuma Caddy config not found"
+    fi
+    
+    echo ""
+}
+
+verify_final_setup() {
+    print_header "Final Verification - Post-Deployment Steps"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "Checking if all post-deployment steps were completed..."
+    echo ""
+    
+    local all_ok=true
+    
+    # Check 1: infra_secrets.yml has Uptime Kuma credentials
+    print_info "Checking infra_secrets.yml..."
+    if grep -q "^uptime_kuma_username:" "$ANSIBLE_DIR/infra_secrets.yml" 2>/dev/null && \
+       grep -q "^uptime_kuma_password:" "$ANSIBLE_DIR/infra_secrets.yml" 2>/dev/null; then
+        local uk_user=$(grep "^uptime_kuma_username:" "$ANSIBLE_DIR/infra_secrets.yml" | awk '{print $2}' | tr -d '"' | tr -d "'")
+        local uk_pass=$(grep "^uptime_kuma_password:" "$ANSIBLE_DIR/infra_secrets.yml" | awk '{print $2}' | tr -d '"' | tr -d "'")
+        
+        if [ -n "$uk_user" ] && [ -n "$uk_pass" ] && [ "$uk_user" != '""' ] && [ "$uk_pass" != '""' ]; then
+            print_success "Uptime Kuma credentials configured in infra_secrets.yml"
+        else
+            print_error "Uptime Kuma credentials are empty in infra_secrets.yml"
+            print_info "Please complete Step 2: Update infra_secrets.yml"
+            all_ok=false
+        fi
+    else
+        print_error "Uptime Kuma credentials not found in infra_secrets.yml"
+        print_info "Please complete Step 2: Update infra_secrets.yml"
+        all_ok=false
+    fi
+    
+    echo ""
+    
+    # Check 2: Can connect to Uptime Kuma API
+    print_info "Checking Uptime Kuma API access..."
+    
+    if [ -n "$uk_user" ] && [ -n "$uk_pass" ]; then
+        # Create a test Python script to check API access
+        local test_script=$(mktemp)
+        cat > "$test_script" << 'EOFPYTHON'
+import sys
+import yaml
+from uptime_kuma_api import UptimeKumaApi
+
+try:
+    # Load config
+    with open('infra_vars.yml', 'r') as f:
+        infra_vars = yaml.safe_load(f)
+    
+    with open('services/uptime_kuma/uptime_kuma_vars.yml', 'r') as f:
+        uk_vars = yaml.safe_load(f)
+    
+    with open('infra_secrets.yml', 'r') as f:
+        secrets = yaml.safe_load(f)
+    
+    root_domain = infra_vars.get('root_domain')
+    subdomain = uk_vars.get('uptime_kuma_subdomain', 'uptime')
+    url = f"https://{subdomain}.{root_domain}"
+    
+    username = secrets.get('uptime_kuma_username')
+    password = secrets.get('uptime_kuma_password')
+    
+    # Try to connect
+    api = UptimeKumaApi(url)
+    api.login(username, password)
+    
+    # Check if we can get monitors
+    monitors = api.get_monitors()
+    
+    print(f"SUCCESS:{len(monitors)}")
+    api.disconnect()
+    sys.exit(0)
+    
+except Exception as e:
+    print(f"ERROR:{str(e)}", file=sys.stderr)
+    sys.exit(1)
+EOFPYTHON
+        
+        local result=$(cd "$ANSIBLE_DIR" && python3 "$test_script" 2>&1)
+        rm -f "$test_script"
+        
+        if echo "$result" | grep -q "^SUCCESS:"; then
+            local monitor_count=$(echo "$result" | grep "^SUCCESS:" | cut -d: -f2)
+            print_success "Successfully connected to Uptime Kuma API"
+            print_info "Current monitors: $monitor_count"
+        else
+            print_error "Cannot connect to Uptime Kuma API"
+            print_warning "This usually means:"
+            echo "  • Admin account not created yet (Step 1)"
+            echo "  • Wrong credentials in infra_secrets.yml (Step 2)"
+            echo "  • Uptime Kuma not accessible"
+            all_ok=false
+        fi
+    else
+        print_warning "Skipping API check - credentials not configured"
+        all_ok=false
+    fi
+    
+    echo ""
+    
+    # Check 3: ntfy notification configured in Uptime Kuma
+    print_info "Checking ntfy notification configuration..."
+    
+    if [ -n "$uk_user" ] && [ -n "$uk_pass" ]; then
+        local test_notif=$(mktemp)
+        cat > "$test_notif" << 'EOFPYTHON'
+import sys
+import yaml
+from uptime_kuma_api import UptimeKumaApi
+
+try:
+    # Load config
+    with open('infra_vars.yml', 'r') as f:
+        infra_vars = yaml.safe_load(f)
+    
+    with open('services/uptime_kuma/uptime_kuma_vars.yml', 'r') as f:
+        uk_vars = yaml.safe_load(f)
+    
+    with open('infra_secrets.yml', 'r') as f:
+        secrets = yaml.safe_load(f)
+    
+    root_domain = infra_vars.get('root_domain')
+    subdomain = uk_vars.get('uptime_kuma_subdomain', 'uptime')
+    url = f"https://{subdomain}.{root_domain}"
+    
+    username = secrets.get('uptime_kuma_username')
+    password = secrets.get('uptime_kuma_password')
+    
+    # Connect
+    api = UptimeKumaApi(url)
+    api.login(username, password)
+    
+    # Check for ntfy notification
+    notifications = api.get_notifications()
+    ntfy_found = any(n.get('type') == 'ntfy' for n in notifications)
+    
+    if ntfy_found:
+        print("SUCCESS:ntfy notification configured")
+    else:
+        print("NOTFOUND:No ntfy notification found")
+    
+    api.disconnect()
+    sys.exit(0)
+    
+except Exception as e:
+    print(f"ERROR:{str(e)}", file=sys.stderr)
+    sys.exit(1)
+EOFPYTHON
+        
+        local notif_result=$(cd "$ANSIBLE_DIR" && python3 "$test_notif" 2>&1)
+        rm -f "$test_notif"
+        
+        if echo "$notif_result" | grep -q "^SUCCESS:"; then
+            print_success "ntfy notification is configured in Uptime Kuma"
+        elif echo "$notif_result" | grep -q "^NOTFOUND:"; then
+            print_warning "ntfy notification not yet configured"
+            print_info "Run the script again and choose 'yes' for ntfy notification setup"
+            print_info "Or complete Step 3 manually"
+            all_ok=false
+        else
+            print_warning "Could not verify ntfy notification (API access issue)"
+        fi
+    else
+        print_warning "Skipping ntfy check - credentials not configured"
+    fi
+    
+    echo ""
+    
+    # Summary
+    if [ "$all_ok" = true ]; then
+        print_success "All post-deployment steps completed! ✓"
+        echo ""
+        print_info "Layer 4 is fully configured and ready to use"
+        print_info "You can now proceed to Layer 6 (infrastructure monitoring)"
+        return 0
+    else
+        print_warning "Some post-deployment steps are incomplete"
+        echo ""
+        print_info "Complete these steps:"
+        echo "  1. Access Uptime Kuma web UI and create admin account"
+        echo "  2. Update ansible/infra_secrets.yml with credentials"
+        echo "  3. Run this script again to configure ntfy notification"
+        echo ""
+        print_info "You can also complete manually and verify with:"
+        echo "  ./scripts/setup_layer_4_monitoring.sh"
+        return 1
+    fi
+}
+
+###############################################################################
+# Summary Functions
+###############################################################################
+
+print_summary() {
+    print_header "Layer 4 Setup Complete! 🎉"
+    
+    echo "Summary of what was configured:"
+    echo ""
+    print_success "ntfy notification service deployed"
+    print_success "Uptime Kuma monitoring platform deployed"
+    print_success "Caddy reverse proxy configured for both services"
+    echo ""
+    
+    print_warning "REQUIRED POST-DEPLOYMENT STEPS:"
+    echo ""
+    echo "MANUAL (do these first):"
+    echo "  1. Access Uptime Kuma Web UI and create admin account"
+    echo "  2. Update ansible/infra_secrets.yml with credentials"
+    echo ""
+    echo "AUTOMATED (script can do these):"
+    echo "  3. Configure ntfy notification - script will offer to set this up"
+    echo "  4. Final verification - script will check everything"
+    echo ""
+    print_info "After completing steps 1 & 2, the script will:"
+    echo "  • Automatically configure ntfy in Uptime Kuma"
+    echo "  • Verify all post-deployment steps"
+    echo "  • Tell you if anything is missing"
+    echo ""
+    print_warning "You MUST complete steps 1 & 2 before proceeding to Layer 6!"
+    echo ""
+    
+    print_info "What these services enable:"
+    echo "  • ntfy: Push notifications to your devices"
+    echo "  • Uptime Kuma: Monitor all services and infrastructure"
+    echo "  • Together: Complete monitoring and alerting solution"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Complete the post-deployment steps above"
+    echo "  2. Test ntfy: Send a test notification"
+    echo "  3. Test Uptime Kuma: Create a test monitor"
+    echo "  4. Proceed to Layer 5: ./scripts/setup_layer_5_headscale.sh (optional)"
+    echo "     OR Layer 6: ./scripts/setup_layer_6_infra_monitoring.sh"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "📊 Layer 4: Core Monitoring & Notifications"
+    
+    echo "This script will deploy ntfy and Uptime Kuma on watchtower."
+    echo ""
+    print_info "Services to deploy:"
+    echo "  • ntfy (notification service)"
+    echo "  • Uptime Kuma (monitoring platform)"
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 4 setup?"; then
+        echo "Setup cancelled."
+        exit 0
+    fi
+    
+    check_prerequisites
+    check_vars_files
+    check_dns_configuration
+    
+    # Deploy services (don't fail if skipped)
+    deploy_ntfy || true
+    echo ""
+    deploy_uptime_kuma || true
+    echo ""
+    setup_uptime_kuma_backup || true
+    
+    echo ""
+    verify_deployments
+    
+    # Always show summary and offer ntfy configuration
+    print_summary
+    echo ""
+    
+    # Always ask about ntfy notification setup (regardless of deployment status)
+    print_header "Configure ntfy Notification in Uptime Kuma"
+    print_info "After creating your Uptime Kuma admin account and updating infra_secrets.yml,"
+    print_info "the script can automatically configure ntfy as a notification method."
+    echo ""
+    print_warning "Prerequisites:"
+    echo "  1. Access Uptime Kuma web UI and create admin account"
+    echo "  2. Update ansible/infra_secrets.yml with your credentials"
+    echo ""
+    
+    # Always offer to set up ntfy notification
+    setup_ntfy_notification
+    
+    # Final verification
+    echo ""
+    verify_final_setup
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_5_headscale.sh

@@ -0,0 +1,524 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 5: VPN Infrastructure (Headscale)
+# 
+# This script deploys Headscale and optionally joins machines to the mesh.
+# Must be run after Layers 0, 1A, and 3 are complete.
+# THIS LAYER IS OPTIONAL - skip to Layer 6 if you don't need VPN.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+check_prerequisites() {
+    print_header "Verifying Prerequisites"
+    
+    local errors=0
+    
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+    
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found"
+    fi
+    
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+    
+    # Check if spacey is configured
+    if [ -z "$(get_hosts_from_inventory "spacey")" ]; then
+        print_error "spacey not configured in inventory.ini"
+        print_info "Layer 5 requires spacey VPS for Headscale server"
+        ((errors++))
+    else
+        print_success "spacey configured in inventory"
+    fi
+    
+    if [ $errors -gt 0 ]; then
+        print_error "Prerequisites not met"
+        exit 1
+    fi
+    
+    print_success "Prerequisites verified"
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+if target in data:
+    print(' '.join(data[target].get('hosts', [])))
+else:
+    hostvars = data.get('_meta', {}).get('hostvars', {})
+    if target in hostvars:
+        print(target)
+PY
+}
+
+get_host_ip() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+hostvars = data.get('_meta', {}).get('hostvars', {})
+if target in hostvars:
+    print(hostvars[target].get('ansible_host', target))
+else:
+    hosts = data.get(target, {}).get('hosts', [])
+    if hosts:
+        first = hosts[0]
+        hv = hostvars.get(first, {})
+        print(hv.get('ansible_host', first))
+PY
+}
+
+check_vars_files() {
+    print_header "Checking Configuration Files"
+    
+    # Check services_config.yml
+    if [ ! -f "$ANSIBLE_DIR/services_config.yml" ]; then
+        print_error "services_config.yml not found"
+        print_info "This file should have been created in Layer 0"
+        exit 1
+    fi
+    
+    print_success "services_config.yml exists"
+    
+    # Show configured subdomain
+    local hs_sub=$(grep "^  headscale:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "headscale")
+    print_info "Configured subdomain: headscale: $hs_sub"
+    echo ""
+}
+
+check_dns_configuration() {
+    print_header "Validating DNS Configuration"
+    
+    cd "$ANSIBLE_DIR"
+    
+    # Get spacey IP
+    local spacey_ip=$(get_host_ip "spacey")
+    
+    if [ -z "$spacey_ip" ]; then
+        print_error "Could not determine spacey IP from inventory"
+        return 1
+    fi
+    
+    print_info "Spacey IP: $spacey_ip"
+    echo ""
+    
+    # Get domain from infra_vars.yml
+    local root_domain=$(grep "^root_domain:" "$ANSIBLE_DIR/infra_vars.yml" | awk '{print $2}' 2>/dev/null)
+    
+    if [ -z "$root_domain" ]; then
+        print_error "Could not determine root_domain from infra_vars.yml"
+        return 1
+    fi
+    
+    # Get subdomain from centralized config
+    local headscale_subdomain="headscale"
+    
+    if [ -f "$ANSIBLE_DIR/services_config.yml" ]; then
+        headscale_subdomain=$(grep "^  headscale:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "headscale")
+    fi
+    
+    local headscale_fqdn="${headscale_subdomain}.${root_domain}"
+    
+    print_info "Checking DNS record..."
+    echo ""
+    
+    # Check Headscale DNS
+    print_info "Checking $headscale_fqdn..."
+    if command -v dig &> /dev/null; then
+        local resolved=$(dig +short "$headscale_fqdn" | head -n1)
+        if [ "$resolved" = "$spacey_ip" ]; then
+            print_success "$headscale_fqdn → $resolved ✓"
+        elif [ -n "$resolved" ]; then
+            print_error "$headscale_fqdn → $resolved (expected $spacey_ip)"
+            print_warning "DNS changes can take time to propagate (up to 24-48 hours)"
+            echo ""
+            if ! confirm_action "Continue anyway? (SSL certificates will fail without proper DNS)"; then
+                exit 1
+            fi
+        else
+            print_error "$headscale_fqdn does not resolve"
+            print_warning "DNS changes can take time to propagate"
+            echo ""
+            if ! confirm_action "Continue anyway? (SSL certificates will fail without proper DNS)"; then
+                exit 1
+            fi
+        fi
+    else
+        print_warning "dig command not found, skipping DNS validation"
+        print_info "Install dnsutils/bind-tools to enable DNS validation"
+    fi
+    
+    echo ""
+    print_success "DNS validation complete"
+}
+
+###############################################################################
+# Headscale Deployment
+###############################################################################
+
+deploy_headscale() {
+    print_header "Deploying Headscale Server"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Install Headscale on spacey"
+    echo "  • Configure with deny-all ACL policy (you customize later)"
+    echo "  • Create namespace for your network"
+    echo "  • Set up Caddy reverse proxy"
+    echo "  • Configure embedded DERP server"
+    echo ""
+    
+    print_warning "After deployment, you MUST configure ACL policies for machines to communicate"
+    echo ""
+    
+    if ! confirm_action "Proceed with Headscale deployment?"; then
+        print_warning "Skipped Headscale deployment"
+        return 1
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/headscale/deploy_headscale_playbook.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/headscale/deploy_headscale_playbook.yml; then
+        print_success "Headscale deployment complete"
+        return 0
+    else
+        print_error "Headscale deployment failed"
+        return 1
+    fi
+}
+
+###############################################################################
+# Join Machines to Mesh
+###############################################################################
+
+join_machines_to_mesh() {
+    print_header "Join Machines to Mesh (Optional)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will install Tailscale client and join machines to your Headscale mesh"
+    echo ""
+    
+    # Show available hosts
+    echo "Available hosts to join:"
+    for group in vipy watchtower nodito lapy; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            echo "  [$group]: $hosts"
+        fi
+    done
+    echo ""
+    
+    print_info "Join options:"
+    echo "  1. Join recommended machines (vipy, watchtower, nodito)"
+    echo "  2. Join all machines"
+    echo "  3. Custom selection (specify groups)"
+    echo "  4. Skip - join machines later manually"
+    echo ""
+    
+    echo -e -n "${BLUE}Choose option${NC} [1-4]: "
+    read option
+    
+    local limit_hosts=""
+    case "$option" in
+        1)
+            limit_hosts="vipy,watchtower,nodito"
+            print_info "Joining: vipy, watchtower, nodito"
+            ;;
+        2)
+            limit_hosts="all"
+            print_info "Joining: all hosts"
+            ;;
+        3)
+            echo -e -n "${BLUE}Enter groups (comma-separated, e.g., vipy,watchtower)${NC}: "
+            read limit_hosts
+            print_info "Joining: $limit_hosts"
+            ;;
+        4)
+            print_warning "Skipping machine join - you can join manually later"
+            print_info "To join manually:"
+            echo "  ansible-playbook -i inventory.ini infra/920_join_headscale_mesh.yml --limit <host>"
+            return 0
+            ;;
+        *)
+            print_error "Invalid option"
+            return 0
+            ;;
+    esac
+    
+    echo ""
+    if ! confirm_action "Proceed with joining machines?"; then
+        print_warning "Skipped joining machines"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini infra/920_join_headscale_mesh.yml --limit $limit_hosts"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini infra/920_join_headscale_mesh.yml --limit "$limit_hosts"; then
+        print_success "Machines joined to mesh"
+        return 0
+    else
+        print_error "Failed to join some machines"
+        print_info "You can retry or join manually later"
+        return 0
+    fi
+}
+
+###############################################################################
+# Backup Configuration
+###############################################################################
+
+setup_headscale_backup() {
+    print_header "Setting Up Headscale Backup (Optional)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will set up automated backups to lapy"
+    echo ""
+    
+    if ! confirm_action "Set up Headscale backup to lapy?"; then
+        print_warning "Skipped backup setup"
+        return 0
+    fi
+    
+    # Check if rsync is available
+    print_info "Verifying rsync is installed on spacey and lapy..."
+    if ! ansible spacey -i inventory.ini -m shell -a "command -v rsync" &>/dev/null; then
+        print_error "rsync not found on spacey"
+        print_info "Run Layer 2 to install rsync"
+        print_warning "Backup setup skipped - rsync not available"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/headscale/setup_backup_headscale_to_lapy.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/headscale/setup_backup_headscale_to_lapy.yml; then
+        print_success "Headscale backup configured"
+        print_info "Backups will run periodically via cron"
+        return 0
+    else
+        print_error "Backup setup failed"
+        return 0
+    fi
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+verify_deployment() {
+    print_header "Verifying Headscale Deployment"
+    
+    cd "$ANSIBLE_DIR"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    local spacey_host=$(get_hosts_from_inventory "spacey")
+    
+    if [ -z "$spacey_host" ]; then
+        print_error "Could not determine spacey host"
+        return
+    fi
+    
+    print_info "Checking Headscale on spacey ($spacey_host)..."
+    echo ""
+    
+    # Check Headscale service
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$spacey_host "systemctl is-active headscale" &>/dev/null; then
+        print_success "Headscale service running"
+    else
+        print_warning "Headscale service not running"
+    fi
+    
+    # Check Caddy config
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$spacey_host "test -f /etc/caddy/sites-enabled/headscale.conf" &>/dev/null; then
+        print_success "Headscale Caddy config exists"
+    else
+        print_warning "Headscale Caddy config not found"
+    fi
+    
+    # Check ACL file
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$spacey_host "test -f /etc/headscale/acl.json" &>/dev/null; then
+        print_success "ACL policy file exists"
+    else
+        print_warning "ACL policy file not found"
+    fi
+    
+    # List nodes
+    print_info "Attempting to list connected nodes..."
+    local nodes_output=$(timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$spacey_host "sudo headscale nodes list" 2>/dev/null || echo "")
+    
+    if [ -n "$nodes_output" ]; then
+        echo "$nodes_output"
+    else
+        print_warning "Could not list nodes (this is normal if no machines joined yet)"
+    fi
+    
+    echo ""
+}
+
+###############################################################################
+# Summary Functions
+###############################################################################
+
+print_summary() {
+    print_header "Layer 5 Setup Complete! 🎉"
+    
+    echo "Summary of what was configured:"
+    echo ""
+    print_success "Headscale VPN server deployed on spacey"
+    print_success "Caddy reverse proxy configured"
+    print_success "Namespace created for your network"
+    echo ""
+    
+    print_warning "CRITICAL POST-DEPLOYMENT STEPS:"
+    echo ""
+    echo "1. Configure ACL Policies (REQUIRED for machines to communicate):"
+    echo "   • SSH to spacey: ssh counterweight@<spacey-ip>"
+    echo "   • Edit ACL: sudo nano /etc/headscale/acl.json"
+    echo "   • Add rules to allow communication"
+    echo "   • Restart: sudo systemctl restart headscale"
+    echo ""
+    echo "2. Verify machines joined (if you selected that option):"
+    echo "   • SSH to spacey: ssh counterweight@<spacey-ip>"
+    echo "   • List nodes: sudo headscale nodes list"
+    echo ""
+    echo "3. Join additional machines (mobile, desktop):"
+    echo "   • Generate key: sudo headscale preauthkeys create --user <namespace> --reusable"
+    echo "   • On device: tailscale up --login-server https://<headscale-domain> --authkey <key>"
+    echo ""
+    
+    print_info "What Headscale enables:"
+    echo "  • Secure mesh networking between all machines"
+    echo "  • Magic DNS - access machines by hostname"
+    echo "  • NAT traversal - works behind firewalls"
+    echo "  • Self-hosted Tailscale alternative"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Configure ACL policies on spacey"
+    echo "  2. Verify nodes are connected"
+    echo "  3. Proceed to Layer 6: ./scripts/setup_layer_6_infra_monitoring.sh"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "🔐 Layer 5: VPN Infrastructure (Headscale)"
+    
+    echo "This script will deploy Headscale for secure mesh networking."
+    echo ""
+    print_warning "THIS LAYER IS OPTIONAL"
+    print_info "Skip to Layer 6 if you don't need VPN mesh networking"
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 5 setup?"; then
+        echo "Setup skipped - proceeding to Layer 6 is fine!"
+        exit 0
+    fi
+    
+    check_prerequisites
+    check_vars_files
+    check_dns_configuration
+    
+    # Deploy Headscale
+    if deploy_headscale; then
+        echo ""
+        join_machines_to_mesh
+        echo ""
+        setup_headscale_backup
+        echo ""
+        verify_deployment
+        print_summary
+    else
+        print_error "Headscale deployment failed"
+        exit 1
+    fi
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_6_infra_monitoring.sh

@@ -0,0 +1,473 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 6: Infrastructure Monitoring
+# 
+# This script deploys disk usage, healthcheck, and CPU temp monitoring.
+# Must be run after Layer 4 (Uptime Kuma) is complete with credentials set.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+check_prerequisites() {
+    print_header "Verifying Prerequisites"
+    
+    local errors=0
+    
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+    
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found"
+    fi
+    
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+    
+    # Check Python uptime-kuma-api
+    if ! python3 -c "import uptime_kuma_api" 2>/dev/null; then
+        print_error "uptime-kuma-api Python package not found"
+        print_info "Install with: pip install -r requirements.txt"
+        ((errors++))
+    else
+        print_success "uptime-kuma-api package found"
+    fi
+    
+    if [ $errors -gt 0 ]; then
+        print_error "Prerequisites not met"
+        exit 1
+    fi
+    
+    print_success "Prerequisites verified"
+}
+
+check_uptime_kuma_credentials() {
+    print_header "Verifying Uptime Kuma Configuration"
+    
+    cd "$ANSIBLE_DIR"
+    
+    # Check if infra_secrets.yml has credentials
+    if ! grep -q "^uptime_kuma_username:" "$ANSIBLE_DIR/infra_secrets.yml" 2>/dev/null || \
+       ! grep -q "^uptime_kuma_password:" "$ANSIBLE_DIR/infra_secrets.yml" 2>/dev/null; then
+        print_error "Uptime Kuma credentials not found in infra_secrets.yml"
+        print_info "You must complete Layer 4 post-deployment steps first:"
+        echo "  1. Create admin user in Uptime Kuma web UI"
+        echo "  2. Add credentials to ansible/infra_secrets.yml"
+        exit 1
+    fi
+    
+    local uk_user=$(grep "^uptime_kuma_username:" "$ANSIBLE_DIR/infra_secrets.yml" | awk '{print $2}' | tr -d '"' | tr -d "'")
+    local uk_pass=$(grep "^uptime_kuma_password:" "$ANSIBLE_DIR/infra_secrets.yml" | awk '{print $2}' | tr -d '"' | tr -d "'")
+    
+    if [ -z "$uk_user" ] || [ -z "$uk_pass" ]; then
+        print_error "Uptime Kuma credentials are empty in infra_secrets.yml"
+        exit 1
+    fi
+    
+    print_success "Uptime Kuma credentials found"
+    
+    # Test API connection
+    print_info "Testing Uptime Kuma API connection..."
+    
+    local test_script=$(mktemp)
+    cat > "$test_script" << 'EOFPYTHON'
+import sys
+import yaml
+from uptime_kuma_api import UptimeKumaApi
+
+try:
+    with open('infra_vars.yml', 'r') as f:
+        infra_vars = yaml.safe_load(f)
+    
+    with open('services_config.yml', 'r') as f:
+        services_config = yaml.safe_load(f)
+    
+    with open('infra_secrets.yml', 'r') as f:
+        secrets = yaml.safe_load(f)
+    
+    root_domain = infra_vars.get('root_domain')
+    subdomain = services_config.get('subdomains', {}).get('uptime_kuma', 'uptime')
+    url = f"https://{subdomain}.{root_domain}"
+    
+    username = secrets.get('uptime_kuma_username')
+    password = secrets.get('uptime_kuma_password')
+    
+    api = UptimeKumaApi(url)
+    api.login(username, password)
+    
+    monitors = api.get_monitors()
+    print(f"SUCCESS:{len(monitors)}")
+    api.disconnect()
+    
+except Exception as e:
+    print(f"ERROR:{str(e)}", file=sys.stderr)
+    sys.exit(1)
+EOFPYTHON
+    
+    local result=$(cd "$ANSIBLE_DIR" && python3 "$test_script" 2>&1)
+    rm -f "$test_script"
+    
+    if echo "$result" | grep -q "^SUCCESS:"; then
+        local monitor_count=$(echo "$result" | grep "^SUCCESS:" | cut -d: -f2)
+        print_success "Successfully connected to Uptime Kuma API"
+        print_info "Current monitors: $monitor_count"
+    else
+        print_error "Cannot connect to Uptime Kuma API"
+        print_info "Error: $result"
+        echo ""
+        print_info "Make sure:"
+        echo "  • Uptime Kuma is running (Layer 4)"
+        echo "  • Credentials are correct in infra_secrets.yml"
+        echo "  • Uptime Kuma is accessible"
+        exit 1
+    fi
+    
+    echo ""
+    print_success "Uptime Kuma configuration verified"
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+if target in data:
+    print(' '.join(data[target].get('hosts', [])))
+else:
+    hostvars = data.get('_meta', {}).get('hostvars', {})
+    if target in hostvars:
+        print(target)
+PY
+}
+
+###############################################################################
+# Disk Usage Monitoring
+###############################################################################
+
+deploy_disk_usage_monitoring() {
+    print_header "Deploying Disk Usage Monitoring"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will deploy disk usage monitoring on selected hosts"
+    print_info "Default settings:"
+    echo "  • Threshold: 80%"
+    echo "  • Check interval: 15 minutes"
+    echo "  • Mount point: /"
+    echo ""
+    
+    # Show available hosts
+    echo "Available hosts:"
+    for group in vipy watchtower spacey nodito lapy; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            echo "  [$group]: $hosts"
+        fi
+    done
+    echo ""
+    
+    print_info "Deployment options:"
+    echo "  1. Deploy on all remote hosts (vipy, watchtower, spacey, nodito)"
+    echo "  2. Deploy on all hosts (including lapy)"
+    echo "  3. Custom selection (specify groups)"
+    echo "  4. Skip disk monitoring"
+    echo ""
+    
+    echo -e -n "${BLUE}Choose option${NC} [1-4]: "
+    read option
+    
+    local limit_hosts=""
+    case "$option" in
+        1)
+            limit_hosts="vipy,watchtower,spacey,nodito"
+            print_info "Deploying to remote hosts"
+            ;;
+        2)
+            limit_hosts="all"
+            print_info "Deploying to all hosts"
+            ;;
+        3)
+            echo -e -n "${BLUE}Enter groups (comma-separated)${NC}: "
+            read limit_hosts
+            print_info "Deploying to: $limit_hosts"
+            ;;
+        4)
+            print_warning "Skipping disk usage monitoring"
+            return 0
+            ;;
+        *)
+            print_error "Invalid option"
+            return 0
+            ;;
+    esac
+    
+    echo ""
+    if ! confirm_action "Proceed with disk usage monitoring deployment?"; then
+        print_warning "Skipped"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini infra/410_disk_usage_alerts.yml --limit $limit_hosts"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini infra/410_disk_usage_alerts.yml --limit "$limit_hosts"; then
+        print_success "Disk usage monitoring deployed"
+        return 0
+    else
+        print_error "Deployment failed"
+        return 0
+    fi
+}
+
+###############################################################################
+# System Healthcheck Monitoring
+###############################################################################
+
+deploy_system_healthcheck() {
+    print_header "Deploying System Healthcheck Monitoring"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will deploy system healthcheck monitoring on selected hosts"
+    print_info "Default settings:"
+    echo "  • Heartbeat interval: 60 seconds"
+    echo "  • Upside-down mode (no news is good news)"
+    echo ""
+    
+    # Show available hosts
+    echo "Available hosts:"
+    for group in vipy watchtower spacey nodito lapy; do
+        local hosts=$(get_hosts_from_inventory "$group")
+        if [ -n "$hosts" ]; then
+            echo "  [$group]: $hosts"
+        fi
+    done
+    echo ""
+    
+    print_info "Deployment options:"
+    echo "  1. Deploy on all remote hosts (vipy, watchtower, spacey, nodito)"
+    echo "  2. Deploy on all hosts (including lapy)"
+    echo "  3. Custom selection (specify groups)"
+    echo "  4. Skip healthcheck monitoring"
+    echo ""
+    
+    echo -e -n "${BLUE}Choose option${NC} [1-4]: "
+    read option
+    
+    local limit_hosts=""
+    case "$option" in
+        1)
+            limit_hosts="vipy,watchtower,spacey,nodito"
+            print_info "Deploying to remote hosts"
+            ;;
+        2)
+            limit_hosts="all"
+            print_info "Deploying to all hosts"
+            ;;
+        3)
+            echo -e -n "${BLUE}Enter groups (comma-separated)${NC}: "
+            read limit_hosts
+            print_info "Deploying to: $limit_hosts"
+            ;;
+        4)
+            print_warning "Skipping healthcheck monitoring"
+            return 0
+            ;;
+        *)
+            print_error "Invalid option"
+            return 0
+            ;;
+    esac
+    
+    echo ""
+    if ! confirm_action "Proceed with healthcheck monitoring deployment?"; then
+        print_warning "Skipped"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini infra/420_system_healthcheck.yml --limit $limit_hosts"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini infra/420_system_healthcheck.yml --limit "$limit_hosts"; then
+        print_success "System healthcheck monitoring deployed"
+        return 0
+    else
+        print_error "Deployment failed"
+        return 0
+    fi
+}
+
+###############################################################################
+# CPU Temperature Monitoring (Nodito)
+###############################################################################
+
+deploy_cpu_temp_monitoring() {
+    print_header "Deploying CPU Temperature Monitoring (Nodito)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    # Check if nodito is configured
+    local nodito_hosts=$(get_hosts_from_inventory "nodito")
+    if [ -z "$nodito_hosts" ]; then
+        print_info "Nodito not configured in inventory, skipping CPU temp monitoring"
+        return 0
+    fi
+    
+    print_info "This will deploy CPU temperature monitoring on nodito (Proxmox)"
+    print_info "Default settings:"
+    echo "  • Threshold: 80°C"
+    echo "  • Check interval: 60 seconds"
+    echo ""
+    
+    echo ""
+    if ! confirm_action "Proceed with CPU temp monitoring deployment?"; then
+        print_warning "Skipped"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini infra/430_cpu_temp_alerts.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini infra/430_cpu_temp_alerts.yml; then
+        print_success "CPU temperature monitoring deployed"
+        return 0
+    else
+        print_error "Deployment failed"
+        return 0
+    fi
+}
+
+###############################################################################
+# Summary
+###############################################################################
+
+print_summary() {
+    print_header "Layer 6 Setup Complete! 🎉"
+    
+    echo "Summary of what was deployed:"
+    echo ""
+    print_success "Infrastructure monitoring configured"
+    print_success "Monitors created in Uptime Kuma"
+    print_success "Systemd services and timers running"
+    echo ""
+    
+    print_info "What you have now:"
+    echo "  • Disk usage monitoring on selected hosts"
+    echo "  • System healthcheck monitoring"
+    echo "  • CPU temperature monitoring (if nodito configured)"
+    echo "  • All organized in host-specific groups"
+    echo ""
+    
+    print_info "Verify your monitoring:"
+    echo "  1. Open Uptime Kuma web UI"
+    echo "  2. Check monitors organized by host groups"
+    echo "  3. Verify monitors are receiving data"
+    echo "  4. Configure notification rules"
+    echo "  5. Watch for alerts via ntfy"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Customize thresholds if needed"
+    echo "  2. Proceed to Layer 7: Core Services deployment"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "📊 Layer 6: Infrastructure Monitoring"
+    
+    echo "This script will deploy automated monitoring for your infrastructure."
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 6 setup?"; then
+        echo "Setup cancelled."
+        exit 0
+    fi
+    
+    check_prerequisites
+    check_uptime_kuma_credentials
+    
+    # Deploy monitoring
+    deploy_disk_usage_monitoring
+    echo ""
+    deploy_system_healthcheck
+    echo ""
+    deploy_cpu_temp_monitoring
+    
+    echo ""
+    print_summary
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_7_services.sh

@@ -0,0 +1,524 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 7: Core Services
+# 
+# This script deploys Vaultwarden, Forgejo, and LNBits on vipy.
+# Must be run after Layers 0, 1A, 2, and 3 are complete.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project root directory
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+###############################################################################
+# Helper Functions
+###############################################################################
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+    
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+###############################################################################
+# Verification Functions
+###############################################################################
+
+check_prerequisites() {
+    print_header "Verifying Prerequisites"
+    
+    local errors=0
+    
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+    
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found"
+    fi
+    
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+    
+    # Check if vipy is configured
+    if [ -z "$(get_hosts_from_inventory "vipy")" ]; then
+        print_error "vipy not configured in inventory.ini"
+        print_info "Layer 7 requires vipy VPS"
+        ((errors++))
+    else
+        print_success "vipy configured in inventory"
+    fi
+    
+    if [ $errors -gt 0 ]; then
+        print_error "Prerequisites not met"
+        exit 1
+    fi
+    
+    print_success "Prerequisites verified"
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+if target in data:
+    print(' '.join(data[target].get('hosts', [])))
+else:
+    hostvars = data.get('_meta', {}).get('hostvars', {})
+    if target in hostvars:
+        print(target)
+PY
+}
+
+get_host_ip() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+hostvars = data.get('_meta', {}).get('hostvars', {})
+if target in hostvars:
+    print(hostvars[target].get('ansible_host', target))
+else:
+    hosts = data.get(target, {}).get('hosts', [])
+    if hosts:
+        first = hosts[0]
+        hv = hostvars.get(first, {})
+        print(hv.get('ansible_host', first))
+PY
+}
+
+check_dns_configuration() {
+    print_header "Validating DNS Configuration"
+    
+    cd "$ANSIBLE_DIR"
+    
+    # Get vipy IP
+    local vipy_ip=$(get_host_ip "vipy")
+    
+    if [ -z "$vipy_ip" ]; then
+        print_error "Could not determine vipy IP from inventory"
+        return 1
+    fi
+    
+    print_info "Vipy IP: $vipy_ip"
+    echo ""
+    
+    # Get domain from infra_vars.yml
+    local root_domain=$(grep "^root_domain:" "$ANSIBLE_DIR/infra_vars.yml" | awk '{print $2}' 2>/dev/null)
+    
+    if [ -z "$root_domain" ]; then
+        print_error "Could not determine root_domain from infra_vars.yml"
+        return 1
+    fi
+    
+    # Get subdomains from centralized config
+    local vw_subdomain="vault"
+    local fg_subdomain="git"
+    local ln_subdomain="lnbits"
+    
+    if [ -f "$ANSIBLE_DIR/services_config.yml" ]; then
+        vw_subdomain=$(grep "^  vaultwarden:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "vault")
+        fg_subdomain=$(grep "^  forgejo:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "git")
+        ln_subdomain=$(grep "^  lnbits:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "lnbits")
+    fi
+    
+    print_info "Checking DNS records..."
+    echo ""
+    
+    local dns_ok=true
+    
+    if command -v dig &> /dev/null; then
+        # Check each subdomain
+        for service in "vaultwarden:$vw_subdomain" "forgejo:$fg_subdomain" "lnbits:$ln_subdomain"; do
+            local name=$(echo "$service" | cut -d: -f1)
+            local subdomain=$(echo "$service" | cut -d: -f2)
+            local fqdn="${subdomain}.${root_domain}"
+            
+            print_info "Checking $fqdn..."
+            local resolved=$(dig +short "$fqdn" | head -n1)
+            
+            if [ "$resolved" = "$vipy_ip" ]; then
+                print_success "$fqdn → $resolved ✓"
+            elif [ -n "$resolved" ]; then
+                print_error "$fqdn → $resolved (expected $vipy_ip)"
+                dns_ok=false
+            else
+                print_error "$fqdn does not resolve"
+                dns_ok=false
+            fi
+        done
+    else
+        print_warning "dig command not found, skipping DNS validation"
+        print_info "Install dnsutils/bind-tools to enable DNS validation"
+        return 1
+    fi
+    
+    echo ""
+    
+    if [ "$dns_ok" = false ]; then
+        print_error "DNS validation failed"
+        print_info "Please configure DNS records for all services"
+        echo ""
+        print_warning "DNS changes can take time to propagate"
+        echo ""
+        if ! confirm_action "Continue anyway? (SSL certificates will fail without proper DNS)"; then
+            exit 1
+        fi
+    else
+        print_success "DNS validation passed"
+    fi
+}
+
+###############################################################################
+# Service Deployment
+###############################################################################
+
+deploy_vaultwarden() {
+    print_header "Deploying Vaultwarden (Password Manager)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Deploy Vaultwarden via Docker"
+    echo "  • Configure Caddy reverse proxy"
+    echo "  • Set up fail2ban protection"
+    echo "  • Enable sign-ups (disable after first user)"
+    echo ""
+    
+    if ! confirm_action "Proceed with Vaultwarden deployment?"; then
+        print_warning "Skipped Vaultwarden deployment"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/vaultwarden/deploy_vaultwarden_playbook.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/vaultwarden/deploy_vaultwarden_playbook.yml; then
+        print_success "Vaultwarden deployed"
+        echo ""
+        print_warning "POST-DEPLOYMENT:"
+        echo "  1. Visit your Vaultwarden subdomain"
+        echo "  2. Create your first user account"
+        echo "  3. Run: ansible-playbook -i inventory.ini services/vaultwarden/disable_vaultwarden_sign_ups_playbook.yml"
+        return 0
+    else
+        print_error "Vaultwarden deployment failed"
+        return 0
+    fi
+}
+
+deploy_forgejo() {
+    print_header "Deploying Forgejo (Git Server)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Install Forgejo binary"
+    echo "  • Create git user and directories"
+    echo "  • Configure Caddy reverse proxy"
+    echo "  • Enable SSH cloning on port 22"
+    echo ""
+    
+    if ! confirm_action "Proceed with Forgejo deployment?"; then
+        print_warning "Skipped Forgejo deployment"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/forgejo/deploy_forgejo_playbook.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/forgejo/deploy_forgejo_playbook.yml; then
+        print_success "Forgejo deployed"
+        echo ""
+        print_warning "POST-DEPLOYMENT:"
+        echo "  1. Visit your Forgejo subdomain"
+        echo "  2. Create admin account on first visit"
+        echo "  3. Add your SSH key for git cloning"
+        return 0
+    else
+        print_error "Forgejo deployment failed"
+        return 0
+    fi
+}
+
+deploy_lnbits() {
+    print_header "Deploying LNBits (Lightning Wallet)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "This will:"
+    echo "  • Install system dependencies and uv (Python 3.12 tooling)"
+    echo "  • Clone LNBits repository (version v1.3.1)"
+    echo "  • Sync dependencies with uv targeting Python 3.12"
+    echo "  • Configure with FakeWallet (testing)"
+    echo "  • Create systemd service"
+    echo "  • Configure Caddy reverse proxy"
+    echo ""
+    
+    if ! confirm_action "Proceed with LNBits deployment?"; then
+        print_warning "Skipped LNBits deployment"
+        return 0
+    fi
+    
+    print_info "Running: ansible-playbook -i inventory.ini services/lnbits/deploy_lnbits_playbook.yml"
+    echo ""
+    
+    if ansible-playbook -i inventory.ini services/lnbits/deploy_lnbits_playbook.yml; then
+        print_success "LNBits deployed"
+        echo ""
+        print_warning "POST-DEPLOYMENT:"
+        echo "  1. Visit your LNBits subdomain"
+        echo "  2. Create superuser on first visit"
+        echo "  3. Configure real Lightning backend (FakeWallet is for testing only)"
+        echo "  4. Disable new user registration"
+        return 0
+    else
+        print_error "LNBits deployment failed"
+        return 0
+    fi
+}
+
+###############################################################################
+# Backup Configuration
+###############################################################################
+
+setup_backups() {
+    print_header "Setting Up Backups (Optional)"
+    
+    cd "$ANSIBLE_DIR"
+    
+    print_info "Configure automated backups to lapy"
+    echo ""
+    
+    # Vaultwarden backup
+    if confirm_action "Set up Vaultwarden backup to lapy?"; then
+        print_info "Running: ansible-playbook -i inventory.ini services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml"
+        if ansible-playbook -i inventory.ini services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml; then
+            print_success "Vaultwarden backup configured"
+        else
+            print_error "Vaultwarden backup setup failed"
+        fi
+        echo ""
+    fi
+    
+    # LNBits backup
+    if confirm_action "Set up LNBits backup to lapy (GPG encrypted)?"; then
+        print_info "Running: ansible-playbook -i inventory.ini services/lnbits/setup_backup_lnbits_to_lapy.yml"
+        if ansible-playbook -i inventory.ini services/lnbits/setup_backup_lnbits_to_lapy.yml; then
+            print_success "LNBits backup configured"
+        else
+            print_error "LNBits backup setup failed"
+        fi
+        echo ""
+    fi
+    
+    print_warning "Forgejo backups are not automated - set up manually if needed"
+}
+
+###############################################################################
+# Verification
+###############################################################################
+
+verify_services() {
+    print_header "Verifying Service Deployments"
+    
+    cd "$ANSIBLE_DIR"
+    
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+    
+    local vipy_host=$(get_hosts_from_inventory "vipy")
+    
+    if [ -z "$vipy_host" ]; then
+        print_error "Could not determine vipy host"
+        return
+    fi
+    
+    print_info "Checking services on vipy ($vipy_host)..."
+    echo ""
+    
+    # Check Vaultwarden
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$vipy_host "docker ps | grep vaultwarden" &>/dev/null; then
+        print_success "Vaultwarden container running"
+    else
+        print_warning "Vaultwarden container not running (may not be deployed)"
+    fi
+    
+    # Check Forgejo
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$vipy_host "systemctl is-active forgejo" &>/dev/null; then
+        print_success "Forgejo service running"
+    else
+        print_warning "Forgejo service not running (may not be deployed)"
+    fi
+    
+    # Check LNBits
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$vipy_host "systemctl is-active lnbits" &>/dev/null; then
+        print_success "LNBits service running"
+    else
+        print_warning "LNBits service not running (may not be deployed)"
+    fi
+    
+    # Check Caddy configs
+    if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$vipy_host "ls /etc/caddy/sites-enabled/*.conf 2>/dev/null" &>/dev/null; then
+        print_success "Caddy configs exist"
+        local configs=$(timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$vipy_host "ls /etc/caddy/sites-enabled/*.conf 2>/dev/null" | xargs -n1 basename)
+        print_info "Configured services:"
+        echo "$configs" | sed 's/^/    /'
+    else
+        print_warning "No Caddy configs found"
+    fi
+    
+    echo ""
+}
+
+###############################################################################
+# Summary
+###############################################################################
+
+print_summary() {
+    print_header "Layer 7 Setup Complete! 🎉"
+    
+    echo "Summary of what was deployed:"
+    echo ""
+    print_success "Core services deployed on vipy"
+    echo ""
+    
+    print_warning "CRITICAL POST-DEPLOYMENT STEPS:"
+    echo ""
+    echo "For each service you deployed, you MUST:"
+    echo ""
+    
+    echo "1. Vaultwarden (if deployed):"
+    echo "   • Visit web UI and create first user"
+    echo "   • Disable sign-ups: ansible-playbook -i inventory.ini services/vaultwarden/disable_vaultwarden_sign_ups_playbook.yml"
+    echo "   • Optional: Set up backup"
+    echo ""
+    
+    echo "2. Forgejo (if deployed):"
+    echo "   • Visit web UI and create admin account"
+    echo "   • Add your SSH public key for git operations"
+    echo "   • Test cloning: git clone git@<forgejo_subdomain>.<yourdomain>:username/repo.git"
+    echo ""
+    
+    echo "3. LNBits (if deployed):"
+    echo "   • Visit web UI and create superuser"
+    echo "   • Configure real Lightning backend (currently FakeWallet)"
+    echo "   • Disable new user registration"
+    echo "   • Optional: Set up encrypted backup"
+    echo ""
+    
+    print_info "Services are now accessible:"
+    echo "  • Vaultwarden: https://<vaultwarden_subdomain>.<yourdomain>"
+    echo "  • Forgejo: https://<forgejo_subdomain>.<yourdomain>"
+    echo "  • LNBits: https://<lnbits_subdomain>.<yourdomain>"
+    echo ""
+    
+    print_success "Uptime Kuma monitors automatically created:"
+    echo "  • Check Uptime Kuma web UI"
+    echo "  • Look in 'services' monitor group"
+    echo "  • Monitors for Vaultwarden, Forgejo, LNBits should appear"
+    echo ""
+    
+    print_info "Next steps:"
+    echo "  1. Complete post-deployment steps above"
+    echo "  2. Test each service"
+    echo "  3. Check Uptime Kuma monitors are working"
+    echo "  4. Proceed to Layer 8: ./scripts/setup_layer_8_secondary_services.sh"
+    echo ""
+}
+
+###############################################################################
+# Main Execution
+###############################################################################
+
+main() {
+    clear
+    
+    print_header "🚀 Layer 7: Core Services"
+    
+    echo "This script will deploy core services on vipy:"
+    echo "  • Vaultwarden (password manager)"
+    echo "  • Forgejo (git server)"
+    echo "  • LNBits (Lightning wallet)"
+    echo ""
+    
+    if ! confirm_action "Continue with Layer 7 setup?"; then
+        echo "Setup cancelled."
+        exit 0
+    fi
+    
+    check_prerequisites
+    check_dns_configuration
+    
+    # Deploy services
+    deploy_vaultwarden
+    echo ""
+    deploy_forgejo
+    echo ""
+    deploy_lnbits
+    
+    echo ""
+    verify_services
+    
+    echo ""
+    setup_backups
+    
+    print_summary
+}
+
+# Run main function
+main "$@"
+

scripts/setup_layer_8_secondary_services.sh

@@ -0,0 +1,384 @@
+#!/bin/bash
+
+###############################################################################
+# Layer 8: Secondary Services
+#
+# This script deploys the ntfy-emergency-app and memos services.
+# Must be run after Layers 0-7 are complete.
+###############################################################################
+
+set -e  # Exit on error
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Project directories
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+ANSIBLE_DIR="$PROJECT_ROOT/ansible"
+
+declare -a LAYER_SUMMARY=()
+
+print_header() {
+    echo -e "\n${BLUE}========================================${NC}"
+    echo -e "${BLUE}$1${NC}"
+    echo -e "${BLUE}========================================${NC}\n"
+}
+
+print_success() {
+    echo -e "${GREEN}✓${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}✗${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}⚠${NC} $1"
+}
+
+print_info() {
+    echo -e "${BLUE}ℹ${NC} $1"
+}
+
+confirm_action() {
+    local prompt="$1"
+    local response
+
+    read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
+    [[ "$response" =~ ^[Yy]$ ]]
+}
+
+record_summary() {
+    LAYER_SUMMARY+=("$1")
+}
+
+get_hosts_from_inventory() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+if target in data:
+    print(' '.join(data[target].get('hosts', [])))
+else:
+    hostvars = data.get('_meta', {}).get('hostvars', {})
+    if target in hostvars:
+        print(target)
+PY
+}
+
+get_primary_host_ip() {
+    local target="$1"
+    cd "$ANSIBLE_DIR"
+    ansible-inventory -i inventory.ini --list | \
+        python3 - "$target" <<'PY' 2>/dev/null || echo ""
+import json, sys
+data = json.load(sys.stdin)
+target = sys.argv[1]
+hostvars = data.get('_meta', {}).get('hostvars', {})
+if target in hostvars:
+    print(hostvars[target].get('ansible_host', target))
+else:
+    hosts = data.get(target, {}).get('hosts', [])
+    if hosts:
+        first = hosts[0]
+        hv = hostvars.get(first, {})
+        print(hv.get('ansible_host', first))
+PY
+}
+
+check_prerequisites() {
+    print_header "Verifying Prerequisites"
+
+    local errors=0
+
+    if [ -z "$VIRTUAL_ENV" ]; then
+        print_error "Virtual environment not activated"
+        echo "Run: source venv/bin/activate"
+        ((errors++))
+    else
+        print_success "Virtual environment activated"
+    fi
+
+    if ! command -v ansible &> /dev/null; then
+        print_error "Ansible not found"
+        ((errors++))
+    else
+        print_success "Ansible found"
+    fi
+
+    if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
+        print_error "inventory.ini not found"
+        ((errors++))
+    else
+        print_success "inventory.ini exists"
+    fi
+
+    if [ ! -f "$ANSIBLE_DIR/infra_vars.yml" ]; then
+        print_error "infra_vars.yml not found"
+        ((errors++))
+    else
+        print_success "infra_vars.yml exists"
+    fi
+
+    if [ ! -f "$ANSIBLE_DIR/services_config.yml" ]; then
+        print_error "services_config.yml not found"
+        ((errors++))
+    else
+        print_success "services_config.yml exists"
+    fi
+
+    if [ -z "$(get_hosts_from_inventory "vipy")" ]; then
+        print_error "vipy not configured in inventory.ini"
+        ((errors++))
+    else
+        print_success "vipy configured in inventory"
+    fi
+
+    if [ -z "$(get_hosts_from_inventory "memos-box")" ]; then
+        print_warning "memos-box not configured in inventory.ini (memos deployment will be skipped)"
+    else
+        print_success "memos-box configured in inventory"
+    fi
+
+    if [ $errors -gt 0 ]; then
+        print_error "Prerequisites not met. Resolve the issues above and re-run the script."
+        exit 1
+    fi
+
+    print_success "Prerequisites verified"
+
+    # Display configured subdomains
+    local emergency_subdomain=$(grep "^  ntfy_emergency_app:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "emergency")
+    local memos_subdomain=$(grep "^  memos:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "memos")
+
+    print_info "Configured subdomains:"
+    echo "  • ntfy_emergency_app: $emergency_subdomain"
+    echo "  • memos: $memos_subdomain"
+    echo ""
+}
+
+check_dns_configuration() {
+    print_header "Validating DNS Configuration"
+
+    if ! command -v dig &> /dev/null; then
+        print_warning "dig command not found. Skipping DNS validation."
+        print_info "Install dnsutils/bind-tools to enable DNS validation."
+        return 0
+    fi
+
+    cd "$ANSIBLE_DIR"
+
+    local root_domain
+    root_domain=$(grep "^root_domain:" "$ANSIBLE_DIR/infra_vars.yml" | awk '{print $2}' 2>/dev/null)
+
+    if [ -z "$root_domain" ]; then
+        print_error "Could not determine root_domain from infra_vars.yml"
+        return 1
+    fi
+
+    local emergency_subdomain=$(grep "^  ntfy_emergency_app:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "emergency")
+    local memos_subdomain=$(grep "^  memos:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "memos")
+
+    local vipy_ip
+    vipy_ip=$(get_primary_host_ip "vipy")
+
+    if [ -z "$vipy_ip" ]; then
+        print_error "Unable to determine vipy IP from inventory"
+        return 1
+    fi
+
+    local memos_ip=""
+    local memos_host=$(get_hosts_from_inventory "memos-box")
+    if [ -n "$memos_host" ]; then
+        memos_ip=$(get_primary_host_ip "$memos_host")
+    fi
+
+    local dns_ok=true
+
+    local emergency_fqdn="${emergency_subdomain}.${root_domain}"
+    local memos_fqdn="${memos_subdomain}.${root_domain}"
+
+    print_info "Expected DNS:"
+    echo "  • $emergency_fqdn → $vipy_ip"
+    if [ -n "$memos_ip" ]; then
+        echo "  • $memos_fqdn → $memos_ip"
+    else
+        echo "  • $memos_fqdn → (skipped - memos-box not in inventory)"
+    fi
+    echo ""
+
+    local resolved
+
+    print_info "Checking $emergency_fqdn..."
+    resolved=$(dig +short "$emergency_fqdn" | head -n1)
+    if [ "$resolved" = "$vipy_ip" ]; then
+        print_success "$emergency_fqdn resolves to $resolved"
+    elif [ -n "$resolved" ]; then
+        print_error "$emergency_fqdn resolves to $resolved (expected $vipy_ip)"
+        dns_ok=false
+    else
+        print_error "$emergency_fqdn does not resolve"
+        dns_ok=false
+    fi
+
+    if [ -n "$memos_ip" ]; then
+        print_info "Checking $memos_fqdn..."
+        resolved=$(dig +short "$memos_fqdn" | head -n1)
+        if [ "$resolved" = "$memos_ip" ]; then
+            print_success "$memos_fqdn resolves to $resolved"
+        elif [ -n "$resolved" ]; then
+            print_error "$memos_fqdn resolves to $resolved (expected $memos_ip)"
+            dns_ok=false
+        else
+            print_error "$memos_fqdn does not resolve"
+            dns_ok=false
+        fi
+    fi
+
+    echo ""
+
+    if [ "$dns_ok" = false ]; then
+        print_error "DNS validation failed."
+        print_info "Update DNS records as shown above and wait for propagation."
+        echo ""
+        if ! confirm_action "Continue anyway? (SSL certificates will fail without correct DNS)"; then
+            exit 1
+        fi
+    else
+        print_success "DNS validation passed"
+    fi
+}
+
+deploy_ntfy_emergency_app() {
+    print_header "Deploying ntfy-emergency-app"
+
+    cd "$ANSIBLE_DIR"
+
+    print_info "This deploys the emergency notification interface pointing at ntfy."
+    echo ""
+
+    if ! confirm_action "Deploy / update the ntfy-emergency-app?"; then
+        print_warning "Skipped ntfy-emergency-app deployment"
+        record_summary "${YELLOW}• ntfy-emergency-app${NC}: skipped"
+        return 0
+    fi
+
+    print_info "Running: ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml"
+    echo ""
+
+    if ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml; then
+        print_success "ntfy-emergency-app deployed successfully"
+        record_summary "${GREEN}• ntfy-emergency-app${NC}: deployed"
+    else
+        print_error "ntfy-emergency-app deployment failed"
+        record_summary "${RED}• ntfy-emergency-app${NC}: failed"
+    fi
+}
+
+deploy_memos() {
+    print_header "Deploying Memos"
+
+    if [ -z "$(get_hosts_from_inventory "memos-box")" ]; then
+        print_warning "memos-box not in inventory. Skipping memos deployment."
+        record_summary "${YELLOW}• memos${NC}: skipped (memos-box missing)"
+        return 0
+    fi
+
+    cd "$ANSIBLE_DIR"
+
+    if ! confirm_action "Deploy / update memos on memos-box?"; then
+        print_warning "Skipped memos deployment"
+        record_summary "${YELLOW}• memos${NC}: skipped"
+        return 0
+    fi
+
+    print_info "Running: ansible-playbook -i inventory.ini services/memos/deploy_memos_playbook.yml"
+    echo ""
+
+    if ansible-playbook -i inventory.ini services/memos/deploy_memos_playbook.yml; then
+        print_success "Memos deployed successfully"
+        record_summary "${GREEN}• memos${NC}: deployed"
+    else
+        print_error "Memos deployment failed"
+        record_summary "${RED}• memos${NC}: failed"
+    fi
+}
+
+verify_services() {
+    print_header "Verifying Deployments"
+
+    cd "$ANSIBLE_DIR"
+
+    local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
+    ssh_key="${ssh_key/#\~/$HOME}"
+
+    local vipy_host
+    vipy_host=$(get_hosts_from_inventory "vipy")
+
+    if [ -n "$vipy_host" ]; then
+        print_info "Checking services on vipy ($vipy_host)..."
+
+        if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$vipy_host "docker ps | grep ntfy-emergency-app" &>/dev/null; then
+            print_success "ntfy-emergency-app container running"
+        else
+            print_warning "ntfy-emergency-app container not running"
+        fi
+
+        echo ""
+    fi
+
+    local memos_host
+    memos_host=$(get_hosts_from_inventory "memos-box")
+    if [ -n "$memos_host" ]; then
+        print_info "Checking memos on memos-box ($memos_host)..."
+        if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$memos_host "systemctl is-active memos" &>/dev/null; then
+            print_success "memos systemd service running"
+        else
+            print_warning "memos systemd service not running"
+        fi
+        echo ""
+    fi
+}
+
+print_summary() {
+    print_header "Layer 8 Summary"
+
+    if [ ${#LAYER_SUMMARY[@]} -eq 0 ]; then
+        print_info "No actions were performed."
+        return
+    fi
+
+    for entry in "${LAYER_SUMMARY[@]}"; do
+        echo -e "$entry"
+    done
+
+    echo ""
+    print_info "Next steps:"
+    echo "  • Visit each service's subdomain to complete any manual setup."
+    echo "  • Configure backups for new services if applicable."
+    echo "  • Update Uptime Kuma monitors if additional endpoints are desired."
+}
+
+main() {
+    print_header "Layer 8: Secondary Services"
+
+    check_prerequisites
+    check_dns_configuration
+
+    deploy_ntfy_emergency_app
+    deploy_memos
+
+    verify_services
+    print_summary
+}
+
+main "$@"
+

tofu/nodito/README.md

@@ -0,0 +1,75 @@
+## Nodito VMs with OpenTofu (Proxmox)
+
+This directory lets you declare VMs on the `nodito` Proxmox node and apply with OpenTofu. It clones the Ansible-built template `debian-13-cloud-init` and places disks on the ZFS pool `proxmox-tank-1`.
+
+### Prereqs
+- Proxmox API token with VM privileges. Example: user `root@pam`, token name `tofu`.
+- OpenTofu installed.
+    ```
+    sudo apt-get update
+    sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
+
+    sudo install -m 0755 -d /etc/apt/keyrings
+    curl -fsSL https://get.opentofu.org/opentofu.gpg | sudo tee /etc/apt/keyrings/opentofu.gpg >/dev/null
+    curl -fsSL https://packages.opentofu.org/opentofu/tofu/gpgkey | sudo gpg --no-tty --batch --dearmor -o /etc/apt/keyrings/opentofu-repo.gpg >/dev/null
+    sudo chmod a+r /etc/apt/keyrings/opentofu.gpg /etc/apt/keyrings/opentofu-repo.gpg
+
+    echo \
+      "deb [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main
+    deb-src [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main" | \
+      sudo tee /etc/apt/sources.list.d/opentofu.list > /dev/null
+    sudo chmod a+r /etc/apt/sources.list.d/opentofu.list
+
+    sudo apt-get update
+    sudo apt-get install -y tofu
+    tofu version
+    ```
+- The Ansible template exists: `debian-13-cloud-init` (VMID 9001 by default).
+
+### Provider Auth
+Create a `terraform.tfvars` (copy from `terraform.tfvars.example`) and set:
+- `proxmox_api_url` (e.g. `https://nodito:8006/api2/json`)
+- `proxmox_api_token_id` (e.g. `root@pam!tofu`)
+- `proxmox_api_token_secret`
+- `ssh_authorized_keys` (your public key content)
+
+Alternatively, you can export env vars and reference them in a tfvars file.
+
+### Declare VMs
+Edit `terraform.tfvars` and fill the `vms` map. Example entry:
+```
+vms = {
+  web1 = {
+    name         = "web1"
+    cores        = 2
+    memory_mb    = 2048
+    disk_size_gb = 20
+    ipconfig0    = "ip=dhcp" # or "ip=192.168.1.50/24,gw=192.168.1.1"
+    data_disks = [
+      {
+        size_gb = 50
+        # storage defaults to var.zfs_storage_name (proxmox-tank-1)
+        # optional: slot = "scsi2"
+      }
+    ]
+  }
+}
+```
+
+All VM disks are created on `zfs_storage_name` (defaults to `proxmox-tank-1`). Network attaches to `vmbr0`. VLAN can be set per-VM with `vlan_tag`.
+
+### Usage
+```
+tofu init
+tofu plan -var-file=terraform.tfvars
+tofu apply -var-file=terraform.tfvars
+```
+
+> VMs are created once and then protected: the module sets `lifecycle.prevent_destroy = true` and ignores subsequent config changes. After the initial apply, manage day‑2 changes directly in Proxmox (or remove the lifecycle block if you need OpenTofu to own ongoing updates).
+
+### Notes
+- Clones are full clones by default (`full_clone = true`).
+- Cloud-init injects `cloud_init_user` and `ssh_authorized_keys`.
+- Disks use `scsi0` on ZFS with `discard` enabled.
+
+

tofu/nodito/main.tf

@@ -0,0 +1,94 @@
+locals {
+  default_ipconfig0 = "ip=dhcp"
+}
+
+resource "proxmox_vm_qemu" "vm" {
+  for_each = var.vms
+
+  name        = each.value.name
+  target_node = var.proxmox_node
+  vmid        = try(each.value.vmid, null)
+
+  onboot     = true
+  agent      = 1
+  clone      = var.template_name
+  full_clone = true
+  vga {
+    type = "serial0"
+  }
+
+  cpu {
+    sockets = 1
+    cores   = each.value.cores
+    type    = "host"
+  }
+  memory = each.value.memory_mb
+
+  scsihw   = "virtio-scsi-pci"
+  boot     = "c"
+  bootdisk = "scsi0"
+
+  lifecycle {
+    prevent_destroy = true
+    ignore_changes = [
+      name,
+      cpu,
+      memory,
+      network,
+      ipconfig0,
+      ciuser,
+      sshkeys,
+      cicustom,
+    ]
+  }
+
+  serial {
+    id   = 0
+    type = "socket"
+  }
+
+  # Network: bridge vmbr0, optional VLAN tag
+  network {
+    id     = 0
+    model  = "virtio"
+    bridge = "vmbr0"
+    tag    = try(each.value.vlan_tag, 0)
+  }
+
+  # Cloud-init: user, ssh keys, IP, and custom snippet for qemu-guest-agent
+  # Note: Using vendor-data snippet (instead of user-data) allows Proxmox to automatically
+  # set the hostname from the VM name. User info is set separately via ciuser/sshkeys.
+  # Using 'local' storage for snippets (not ZFS) as ZFS storage doesn't properly support snippet paths
+  ciuser    = var.cloud_init_user
+  sshkeys   = var.ssh_authorized_keys
+  ipconfig0 = try(each.value.ipconfig0, local.default_ipconfig0)
+  cicustom  = "vendor=local:snippets/user-data-qemu-agent.yaml"
+
+  # Disk on ZFS storage
+  disk {
+    slot    = "scsi0"
+    type    = "disk"
+    storage = var.zfs_storage_name
+    size    = "${each.value.disk_size_gb}G"
+    # optional flags like iothread/ssd/discard differ by provider versions; keep minimal
+  }
+
+  dynamic "disk" {
+    for_each = try(each.value.data_disks, [])
+    content {
+      slot    = try(disk.value.slot, format("scsi%s", tonumber(disk.key) + 1))
+      type    = "disk"
+      storage = try(disk.value.storage, var.zfs_storage_name)
+      size    = "${disk.value.size_gb}G"
+    }
+  }
+
+  # Cloud-init CD-ROM so ipconfig0/sshkeys apply
+  disk {
+    slot    = "ide2"
+    type    = "cloudinit"
+    storage = var.zfs_storage_name
+  }
+}
+
+

tofu/nodito/provider.tf

@@ -0,0 +1,8 @@
+provider "proxmox" {
+  pm_api_url          = var.proxmox_api_url
+  pm_api_token_id     = var.proxmox_api_token_id
+  pm_api_token_secret = var.proxmox_api_token_secret
+  pm_tls_insecure     = true
+}
+
+

tofu/nodito/terraform.tfvars.example

@@ -0,0 +1,40 @@
+proxmox_api_url           = "https://nodito:8006/api2/json"
+proxmox_api_token_id      = "root@pam!tofu"
+proxmox_api_token_secret  = "REPLACE_ME"
+
+proxmox_node = "nodito"
+zfs_storage_name = "proxmox-tank-1"
+template_name = "debian-13-cloud-init"
+cloud_init_user = "counterweight"
+
+# paste your ~/.ssh/id_ed25519.pub or similar
+ssh_authorized_keys = <<EOKEY
+ssh-ed25519 AAAA... your-key
+EOKEY
+
+vms = {
+  web1 = {
+    name         = "web1"
+    vmid         = 1101
+    cores        = 2
+    memory_mb    = 2048
+    disk_size_gb = 20
+    ipconfig0    = "ip=dhcp"
+    data_disks = [
+      {
+        size_gb = 50
+      }
+    ]
+  }
+
+  db1 = {
+    name         = "db1"
+    vmid         = 1102
+    cores        = 4
+    memory_mb    = 4096
+    disk_size_gb = 40
+    ipconfig0    = "ip=dhcp"
+  }
+}
+
+

tofu/nodito/variables.tf

@@ -0,0 +1,67 @@
+variable "proxmox_api_url" {
+  description = "Base URL for Proxmox API, e.g. https://nodito:8006/api2/json"
+  type        = string
+}
+
+variable "proxmox_api_token_id" {
+  description = "Proxmox API token ID, e.g. root@pam!tofu"
+  type        = string
+  sensitive   = true
+}
+
+variable "proxmox_api_token_secret" {
+  description = "Proxmox API token secret"
+  type        = string
+  sensitive   = true
+}
+
+variable "proxmox_node" {
+  description = "Target Proxmox node name"
+  type        = string
+  default     = "nodito"
+}
+
+variable "zfs_storage_name" {
+  description = "Proxmox storage name backed by ZFS (from Ansible: zfs_pool_name)"
+  type        = string
+  default     = "proxmox-tank-1"
+}
+
+variable "template_name" {
+  description = "Cloud-init template to clone (created by Ansible)"
+  type        = string
+  default     = "debian-13-cloud-init"
+}
+
+variable "cloud_init_user" {
+  description = "Default cloud-init user"
+  type        = string
+  default     = "counterweight"
+}
+
+variable "ssh_authorized_keys" {
+  description = "SSH public key content to inject via cloud-init"
+  type        = string
+  sensitive   = true
+}
+
+variable "vms" {
+  description = "Map of VMs to create"
+  type = map(object({
+    name         = string
+    vmid         = optional(number)
+    cores        = number
+    memory_mb    = number
+    disk_size_gb = number
+    vlan_tag     = optional(number)
+    ipconfig0    = optional(string) # e.g. "ip=dhcp" or "ip=192.168.1.50/24,gw=192.168.1.1"
+    data_disks = optional(list(object({
+      size_gb = number
+      storage = optional(string)
+      slot    = optional(string)
+    })), [])
+  }))
+  default = {}
+}
+
+

tofu/nodito/versions.tf

@@ -0,0 +1,12 @@
+terraform {
+  required_version = ">= 1.6.0"
+
+  required_providers {
+    proxmox = {
+      source  = "Telmate/proxmox"
+      version = "= 3.0.2-rc05"
+    }
+  }
+}
+
+