lots of stuff man

01_infra_setup.md

@@ -18,13 +18,13 @@ This describes how to prepare each machine before deploying services on them.
 * Getting and configuring the domain is outside the scope of this repo. Whenever a service needs you to set up a subdomain, it will be mentioned explictly.
 * You should add the domain to the var `root_domain` in `ansible/infra_vars.yml`.
 
-## Prepare the VPSs (vipy and watchtower)
+## Prepare the VPSs (vipy, watchtower and spacey)
 
 ### Source the VPSs
 
 * The guide is agnostic to which provider you pick, but has been tested with VMs from https://99stack.com and contains some operations that are specifically relevant to their VPSs.
 * The expectations are that the VPS ticks the following boxes:
-  + Runs Debian 12 bookworm.
+  + Runs Debian 12/13 bookworm.
   + Has a public IP4 and starts out with SSH listening on port 22.
   + Boots with one of your SSH keys already authorized. If this is not the case, you'll have to manually drop the pubkey there before using the playbooks.
 * You will need three VPSs:
@@ -35,7 +35,7 @@ This describes how to prepare each machine before deploying services on them.
 
 ### Prepare Ansible vars
 
-* You have an example `ansible/example.inventory.ini`. Copy it with `cp ansible/example.inventory.ini ansible/inventory.ini` and fill in with the values for your VPSs. `[vipy]` is the services VPS. `[watchtower]` is the watchtower VPS.
+* You have an example `ansible/example.inventory.ini`. Copy it with `cp ansible/example.inventory.ini ansible/inventory.ini` and fill in with the values for your VPSs. `[vipy]` is the services VPS. `[watchtower]` is the watchtower VPS. `[spacey]`is the headscale VPS.
 * A few notes:
   * The guides assume you'll only have one VPS in the `[vipy]` group. Stuff will break if you have multiple, so avoid that.
 
@@ -131,6 +131,47 @@ Note that, by applying these playbooks, both the root user and the `counterweigh
   * Enable ZFS services for automatic pool import on boot
 * **Warning**: This will destroy all data on the specified disks. Make sure you're using the correct disk IDs and that the disks don't contain important data.
 
+## General prep for all machines
+
+### Set up Infrastructure Secrets
+
+* Create `ansible/infra_secrets.yml` based on the example file:
+  ```bash
+  cp ansible/infra_secrets.yml.example ansible/infra_secrets.yml
+  ```
+* Edit `ansible/infra_secrets.yml` and add your Uptime Kuma credentials:
+  ```yaml
+  uptime_kuma_username: "admin"
+  uptime_kuma_password: "your_password"
+  ```
+* **Important**: Never commit this file to version control (it's in `.gitignore`)
+
+### Deploy Disk Usage Monitoring
+
+* Any machine can be configured with disk usage monitoring that sends alerts to Uptime Kuma when disk usage exceeds a threshold.
+* This playbook automatically creates an Uptime Kuma push monitor for each host (idempotent - won't create duplicates).
+* Prerequisites:
+  * Install the Uptime Kuma Ansible collection: `ansible-galaxy collection install -r ansible/requirements.yml`
+  * Install Python dependencies: `pip install -r requirements.txt` (includes uptime-kuma-api)
+  * Set up `ansible/infra_secrets.yml` with your Uptime Kuma API token (see above)
+  * Uptime Kuma must be deployed (the playbook automatically uses the URL from `uptime_kuma_vars.yml`)
+* Run the disk monitoring setup with:
+  ```bash
+  ansible-playbook -i inventory.ini infra/410_disk_usage_alerts.yml
+  ```
+* This will:
+  * Create an Uptime Kuma monitor group per host named "{hostname} - infra" (idempotent)
+  * Create a push monitor in Uptime Kuma with "upside down" mode (no news is good news)
+  * Assign the monitor to the host's group for better organization
+  * Install required packages (curl, bc)
+  * Create a monitoring script that checks disk usage at configured intervals (default: 15 minutes)
+  * Set up a systemd service and timer for automated monitoring
+  * Send alerts to Uptime Kuma only when usage exceeds threshold (default: 80%)
+* Optional configuration:
+  * Change threshold: `-e "disk_usage_threshold_percent=85"`
+  * Change check interval: `-e "disk_check_interval_minutes=10"`
+  * Monitor different mount point: `-e "monitored_mount_point=/home"`
+
 ## GPG Keys
 
 Some of the backups are stored encrypted for security. To allow this, fill in the gpg variables listed in `example.inventory.ini` under the `lapy` block.