personal_infra/01_infra_setup.md

# 01 Infra Setup

This describes how to prepare each machine before deploying services on them.

## First steps

* Create an ssh key or pick an existing one. We'll refer to it as the `personal_ssh_key`.
* Deploy ansible on the laptop (Lapy), which will act as the ansible control node. To do so:
  * Create a `venv`: `python3 -m venv venv`
  * Activate it: `source venv/bin/activate`
  * Install the listed ansible requirements with `pip install -r requirements.txt`
* Keep in mind you should activate this `venv` from now on when running `ansible` commands.

## Domain

* Some services are designed to be accessible through WAN through a friendly URL.
* You'll need to have a domain where you can set DNS records and have the ability to create different subdomains, as the guide assumes each service will get its own subdomain.
* Getting and configuring the domain is outside the scope of this repo. Whenever a service needs you to set up a subdomain, it will be mentioned explictly.
* You should add the domain to the var `root_domain` in `ansible/infra_vars.yml`.

## Prepare the VPSs (vipy and watchtower)

### Source the VPSs

* The guide is agnostic to which provider you pick, but has been tested with VMs from https://99stack.com and contains some operations that are specifically relevant to their VPSs.
* The expectations are that the VPS ticks the following boxes:
  + Runs Debian 12 bookworm.
  + Has a public IP4 and starts out with SSH listening on port 22.
  + Boots with one of your SSH keys already authorized. If this is not the case, you'll have to manually drop the pubkey there before using the playbooks.
* You will need two VPSs: one to host most services, and another tiny one to monitor Uptime. We use two to prevent the monitoring service from falling down with the main machine.
* Move on once your VPSs are running and satisfies the prerequisites.

### Prepare Ansible vars

* You have an example `ansible/example.inventory.ini`. Copy it with `cp ansible/example.inventory.ini ansible/inventory.ini` and fill in with the values for your VPSs. `[vipy]` is the services VPS. `[watchtower]` is the watchtower VPS.
* A few notes:
  * The guides assume you'll only have one VPS in the `[vipy]` group. Stuff will break if you have multiple, so avoid that.

### Create user and secure VPS access

* Ansible will create a user on the first playbook `01_basic_vps_setup_playbook.yml`. This is the user that will get used regularly. But, since this user doesn't exist, you obviosuly need to first run this playbook from some other user. We assume your VPS provider has given you a root user, which is what you need to define as the running user in the next command.
* cd into `ansible`
* Run `ansible-playbook -i inventory.ini infra/01_user_and_access_setup_playbook.yml -e 'ansible_user="your root user here"'`
* Then, configure firewall access, fail2ban and auditd with `ansible-playbook -i inventory.ini infra/02_firewall_and_fail2ban_playbook.yml`. Since the user we will use is now present, there is no need to specify the user anymore.

Note that, by applying these playbooks, both the root user and the `counterweight` user will use the same SSH pubkey for auth.

## Prepare Nodito Server

### Source the Nodito Server

* This setup is designed for a local Nodito server running in your home environment.
* The expectations are that the Nodito server:
  + Runs Proxmox VE (based on Debian).
  + Has a predictable local IP address.
  + Has root user with password authentication enabled (default Proxmox state).
  + SSH is accessible on port 22.

### Prepare Ansible vars for Nodito

* Add a `[nodito]` group to your `ansible/inventory.ini` (or simply use the one you get by copying `example.inventory.ini`) and fill in with values.

### Bootstrap SSH Key Access and Create User

* Nodito starts with password authentication enabled and no SSH keys configured. We need to bootstrap SSH key access first.
* Run the complete setup with: `ansible-playbook -i inventory.ini infra/nodito/30_proxmox_bootstrap_playbook.yml -e 'ansible_user=root'`
* This single playbook will:
  * Set up SSH key access for root
  * Create the counterweight user with SSH keys
  * Update and secure the system
  * Disable root login and password authentication
  * Test the final configuration
* For all future playbooks targeting nodito, use the default configuration (no overrides needed).

Note that, by applying these playbooks, both the root user and the `counterweight` user will use the same SSH pubkey for auth, but root login will be disabled.

### Switch to Community Repositories

* Proxmox VE installations typically come with enterprise repositories enabled, which require a subscription. To avoid subscription warnings and use the community repositories instead:
* Run the repository switch with: `ansible-playbook -i inventory.ini infra/nodito/32_proxmox_community_repos_playbook.yml`
* This playbook will:
  * Detect whether your Proxmox installation uses modern deb822 format (Proxmox VE 9) or legacy format (Proxmox VE 8)
  * Remove enterprise repository files and create community repository files
  * Disable subscription nag messages in both web and mobile interfaces
  * Update Proxmox packages from the community repository
  * Verify the changes are working correctly
* After running this playbook, clear your browser cache or perform a hard reload (Ctrl+Shift+R) before using the Proxmox VE Web UI to avoid UI display issues.

### Deploy CPU Temperature Monitoring

* The nodito server can be configured with CPU temperature monitoring that sends alerts to Uptime Kuma when temperatures exceed a threshold.
* Before running the CPU temperature monitoring playbook, you need to create a secrets file with your Uptime Kuma push URL:
  * Create `ansible/infra/nodito/nodito_secrets.yml` with:
    ```yaml
    uptime_kuma_url: "https://your-uptime-kuma.com/api/push/your-push-key"
    ```
* Run the CPU temperature monitoring setup with: `ansible-playbook -i inventory.ini infra/nodito/40_cpu_temp_alerts.yml`
* This will:
  * Install required packages (lm-sensors, curl, jq, bc)
  * Create a monitoring script that checks CPU temperature every minute
  * Set up a systemd service and timer for automated monitoring
  * Send alerts to Uptime Kuma when temperature exceeds the threshold (default: 80°C)

### Setup ZFS Storage Pool

* The nodito server can be configured with a ZFS RAID 1 storage pool for Proxmox VM storage, providing redundancy and data integrity.
* Before running the ZFS pool setup playbook, you need to identify your disk IDs and configure them in the variables file:
  * SSH into your nodito server and run: `ls -la /dev/disk/by-id/ | grep -E "(ata-|scsi-|nvme-)"`
  * This will show you the persistent disk identifiers for all your disks. Look for the two disks you want to use for the ZFS pool.
  * Example output:
    ```
    lrwxrwxrwx 1 root root  9 Dec 15 10:30 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1234567 -> ../../sdb
    lrwxrwxrwx 1 root root  9 Dec 15 10:30 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K7654321 -> ../../sdc
    ```
  * Update `ansible/infra/nodito/nodito_vars.yml` with your actual disk IDs:
    ```yaml
    zfs_disk_1: "/dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1234567"
    zfs_disk_2: "/dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K7654321"
    ```
* Run the ZFS pool setup with: `ansible-playbook -i inventory.ini infra/nodito/32_zfs_pool_setup_playbook.yml`
* This will:
  * Validate Proxmox VE and ZFS installation
  * Install ZFS utilities and kernel modules
  * Create a RAID 1 (mirror) ZFS pool named `proxmox-storage` with optimized settings
  * Configure ZFS pool properties (ashift=12, compression=lz4, atime=off, etc.)
  * Export and re-import the pool for Proxmox compatibility
  * Configure Proxmox to use the ZFS pool storage (zfspool type)
  * Enable ZFS services for automatic pool import on boot
* **Warning**: This will destroy all data on the specified disks. Make sure you're using the correct disk IDs and that the disks don't contain important data.

## GPG Keys

Some of the backups are stored encrypted for security. To allow this, fill in the gpg variables listed in `example.inventory.ini` under the `lapy` block.