personal_infra/human_script.md

Personal Infrastructure Setup Guide

This guide walks you through setting up your complete personal infrastructure, layer by layer. Each layer must be completed before moving to the next one.

Automated Setup: Each layer has a bash script that handles the setup process. The scripts will:

• Check prerequisites
• Prompt for required variables
• Set up configuration files
• Execute playbooks
• Verify completion

Prerequisites

Before starting:

• You have a domain name
• You have VPS accounts ready
• You have nodito ready with Proxmox installed, ssh key in place
• You have SSH access to all machines
• You're running this from your laptop (lapy)

---

Layer 0: Foundation Setup

Goal: Set up your laptop (lapy) as the Ansible control node and configure basic settings.

Script: ./scripts/setup_layer_0.sh

What This Layer Does:

1. Creates Python virtual environment
2. Installs Ansible and required Python packages
3. Installs Ansible Galaxy collections
4. Guides you through creating inventory.ini with your machine IPs
5. Guides you through creating infra_vars.yml with your domain
6. Creates services_config.yml with centralized subdomain settings
7. Creates infra_secrets.yml template for Uptime Kuma credentials
8. Validates SSH keys exist
9. Verifies everything is ready for Layer 1

Required Information:

• Your domain name (e.g., contrapeso.xyz)
• SSH key path (default: ~/.ssh/counterganzua)
• IP addresses for your infrastructure:
  • vipy (main VPS)
  • watchtower (monitoring VPS)
  • spacey (headscale VPS)
  • nodito (Proxmox server) - optional
  • Note: VMs (like memos-box) will be created later on Proxmox and added to the nodito-vms group

Manual Steps:

After running the script, you'll need to:

1. Ensure your SSH key is added to all VPS root users (usually done by VPS provider)
2. Ensure DNS is configured for your domain (nameservers pointing to your DNS provider)

Centralized Configuration:

The script creates ansible/services_config.yml which contains all service subdomains in one place:

• Easy to review all subdomains at a glance
• No need to edit multiple vars files
• Consistent Caddy settings across all services
• Edit this file to customize your subdomains before deploying services

Verification:

The script will verify:

• ✓ Python venv exists and activated
• ✓ Ansible installed
• ✓ Required Python packages installed
• ✓ Ansible Galaxy collections installed
• ✓ inventory.ini exists and formatted correctly
• ✓ infra_vars.yml exists with domain configured
• ✓ services_config.yml created with subdomain settings
• ✓ infra_secrets.yml template created
• ✓ SSH key file exists

Run the Script:

cd /home/counterweight/personal_infra
./scripts/setup_layer_0.sh

---

Layer 1A: VPS Basic Setup

Goal: Configure users, SSH access, firewall, and fail2ban on VPS machines.

Script: ./scripts/setup_layer_1a_vps.sh

Can be run independently - doesn't require Nodito setup.

What This Layer Does:

For VPSs (vipy, watchtower, spacey):

1. Creates the counterweight user with sudo access
2. Configures SSH key authentication
3. Disables root login (by design for security)
4. Sets up UFW firewall with SSH access
5. Installs and configures fail2ban
6. Installs and configures auditd for security logging

Prerequisites:

• ✅ Layer 0 complete
• ✅ SSH key added to all VPS root users
• ✅ Root access to VPSs

Verification:

The script will verify:

• ✓ Can SSH to all VPSs as root
• ✓ VPS playbooks complete successfully
• ✓ Can SSH to all VPSs as counterweight user
• ✓ Firewall is active and configured
• ✓ fail2ban is running

Run the Script:

source venv/bin/activate
cd /home/counterweight/personal_infra
./scripts/setup_layer_1a_vps.sh

Note: After this layer, you will no longer be able to SSH as root to VPSs (by design for security).

---

Layer 1B: Nodito (Proxmox) Setup

Goal: Configure the Nodito Proxmox server.

Script: ./scripts/setup_layer_1b_nodito.sh

Can be run independently - doesn't require VPS setup.

What This Layer Does:

For Nodito (Proxmox server):

1. Bootstraps SSH key access for root
2. Creates the counterweight user
3. Updates and secures the system
4. Disables root login and password authentication
5. Switches to Proxmox community repositories
6. Optionally sets up ZFS storage pool (if disks configured)
7. Optionally creates Debian cloud template

Prerequisites:

• ✅ Layer 0 complete
• ✅ Root password for nodito
• ✅ Nodito configured in inventory.ini

Optional: ZFS Setup

For ZFS storage pool (optional):

1. SSH into nodito: ssh root@<nodito-ip>
2. List disk IDs: ls -la /dev/disk/by-id/ | grep -E "(ata-|scsi-|nvme-)"
3. Note the disk IDs you want to use
4. The script will help you create ansible/infra/nodito/nodito_vars.yml with disk configuration

⚠️ Warning: ZFS setup will DESTROY ALL DATA on specified disks!

Verification:

The script will verify:

• ✓ Nodito bootstrap successful
• ✓ Community repos configured
• ✓ Can SSH to nodito as counterweight user

Run the Script:

source venv/bin/activate
cd /home/counterweight/personal_infra
./scripts/setup_layer_1b_nodito.sh

Note: After this layer, you will no longer be able to SSH as root to nodito (by design for security).

---

Layer 2: General Infrastructure Tools

Goal: Install common utilities needed by various services.

Script: ./scripts/setup_layer_2.sh

What This Layer Does:

Installs essential tools on machines that need them:

rsync

• Purpose: Required for backup operations
• Deployed to: vipy, watchtower, lapy (and optionally other hosts)
• Playbook: infra/900_install_rsync.yml

Docker + Docker Compose

• Purpose: Required for containerized services
• Deployed to: vipy, watchtower (and optionally other hosts)
• Playbook: infra/910_docker_playbook.yml

Prerequisites:

• ✅ Layer 0 complete
• ✅ Layer 1A complete (for VPSs) OR Layer 1B complete (for nodito)
• ✅ SSH access as counterweight user

Services That Need These Tools:

• rsync: All backup operations (Uptime Kuma, Vaultwarden, LNBits, etc.)
• docker: Uptime Kuma, Vaultwarden, ntfy-emergency-app

Verification:

The script will verify:

• ✓ rsync installed on specified hosts
• ✓ Docker and Docker Compose installed on specified hosts
• ✓ counterweight user added to docker group
• ✓ Docker service running

Run the Script:

source venv/bin/activate
cd /home/counterweight/personal_infra
./scripts/setup_layer_2.sh

Note: This script is interactive and will let you choose which hosts get which tools.

---

Layer 3: Reverse Proxy (Caddy)

Goal: Deploy Caddy reverse proxy for HTTPS termination and routing.

Script: ./scripts/setup_layer_3_caddy.sh

What This Layer Does:

Installs and configures Caddy web server on VPS machines:

• Installs Caddy from official repositories
• Configures Caddy to listen on ports 80/443
• Opens firewall ports for HTTP/HTTPS
• Creates /etc/caddy/sites-enabled/ directory structure
• Sets up automatic HTTPS with Let's Encrypt

Deployed to: vipy, watchtower, spacey

Why Caddy is Critical:

Caddy provides:

• Automatic HTTPS - Let's Encrypt certificates with auto-renewal
• Reverse proxy - Routes traffic to backend services
• Simple configuration - Each service adds its own config file
• HTTP/2 support - Modern protocol support

Prerequisites:

• ✅ Layer 0 complete
• ✅ Layer 1A complete (VPS setup)
• ✅ SSH access as counterweight user
• ✅ Ports 80/443 available on VPSs

Services That Need Caddy:

All web services depend on Caddy:

• Uptime Kuma (watchtower)
• ntfy (watchtower)
• Headscale (spacey)
• Vaultwarden (vipy)
• Forgejo (vipy)
• LNBits (vipy)
• Personal Blog (vipy)
• ntfy-emergency-app (vipy)

Verification:

The script will verify:

• ✓ Caddy installed on all target hosts
• ✓ Caddy service running
• ✓ Ports 80/443 open in firewall
• ✓ Sites-enabled directory created
• ✓ Can reach Caddy default page

Run the Script:

source venv/bin/activate
cd /home/counterweight/personal_infra
./scripts/setup_layer_3_caddy.sh

Note: Caddy starts with an empty configuration. Services will add their own config files in later layers.

---

Layer 4: Core Monitoring & Notifications

Goal: Deploy ntfy (notifications) and Uptime Kuma (monitoring platform).

Script: ./scripts/setup_layer_4_monitoring.sh

What This Layer Does:

Deploys core monitoring infrastructure on watchtower:

4A: ntfy (Notification Service)

• Installs ntfy from official repositories
• Configures ntfy with authentication (deny-all by default)
• Creates admin user for sending notifications
• Sets up Caddy reverse proxy
• Deployed to: watchtower

4B: Uptime Kuma (Monitoring Platform)

• Deploys Uptime Kuma via Docker
• Configures Caddy reverse proxy
• Sets up data persistence
• Optionally sets up backup to lapy
• Deployed to: watchtower

Prerequisites (Complete BEFORE Running):

1. Previous layers complete:

• ✅ Layer 0, 1A, 2, 3 complete (watchtower must be fully set up)
• ✅ Docker installed on watchtower (from Layer 2)
• ✅ Caddy running on watchtower (from Layer 3)

2. Configure subdomains (in centralized config):

• ✅ Edit ansible/services_config.yml and customize subdomains under subdomains: section
  • Set ntfy: to your preferred subdomain (e.g., ntfy or notify)
  • Set uptime_kuma: to your preferred subdomain (e.g., uptime or kuma)

3. Create DNS records that match your configured subdomains:

• ✅ Create A record: <ntfy_subdomain>.<yourdomain> → watchtower IP
• ✅ Create A record: <uptime_kuma_subdomain>.<yourdomain> → watchtower IP
• ✅ Wait for DNS propagation (can take minutes to hours)
• ✅ Verify with: dig <subdomain>.<yourdomain> should return watchtower IP

4. Prepare ntfy admin credentials:

• ✅ Decide on username (default: admin)
• ✅ Decide on a secure password (script will prompt you)

Run the Script:

source venv/bin/activate
cd /home/counterweight/personal_infra
./scripts/setup_layer_4_monitoring.sh

The script will prompt you for ntfy admin credentials during deployment.

Post-Deployment Steps (Complete AFTER Running):

The script will guide you through most of these, but here's what happens:

Step 1: Set Up Uptime Kuma Admin Account (Manual)

1. Open browser and visit: https://<uptime_kuma_subdomain>.<yourdomain>
2. On first visit, you'll see the setup page
3. Create admin username and password
4. Save these credentials securely

Step 2: Update infra_secrets.yml (Manual)

1. Edit ansible/infra_secrets.yml
2. Add your Uptime Kuma credentials:

   uptime_kuma_username: "your-admin-username"
   uptime_kuma_password: "your-admin-password"
   

3. Save the file
4. This is required for automated ntfy setup and Layer 6

Step 3: Configure ntfy Notification (Automated)

The script will offer to do this automatically! If you completed Steps 1 & 2, the script will:

• Connect to Uptime Kuma via API
• Create ntfy notification configuration
• Test the connection
• No manual UI configuration needed!

Alternatively (Manual):

1. In Uptime Kuma web UI, go to Settings → Notifications
2. Click Setup Notification, choose ntfy
3. Configure with your ntfy subdomain and credentials

Step 4: Final Verification (Automated)

The script will automatically verify:

• ✓ Uptime Kuma credentials in infra_secrets.yml
• ✓ Can connect to Uptime Kuma API
• ✓ ntfy notification is configured
• ✓ All post-deployment steps complete

If anything is missing, the script will tell you exactly what to do!

Step 5: Subscribe to Notifications on Your Phone (Optional - Manual)

1. Install ntfy app: https://github.com/binwiederhier/ntfy-android
2. Add subscription:
   • Server: https://<ntfy_subdomain>.<yourdomain>
   • Topic: alerts (same as configured in Uptime Kuma)
   • Username: Your ntfy admin username
   • Password: Your ntfy admin password
3. You'll now receive push notifications for all alerts!

Pro tip: Run the script again after completing Steps 1 & 2, and it will automatically configure ntfy and verify everything!

Verification:

The script will automatically verify:

• ✓ DNS records are configured correctly (using dig)
• ✓ ntfy service running
• ✓ Uptime Kuma container running
• ✓ Caddy configs created for both services

After post-deployment steps, you can test:

• Visit https://<ntfy_subdomain>.<yourdomain> (should load ntfy web UI)
• Visit https://<uptime_kuma_subdomain>.<yourdomain> (should load Uptime Kuma)
• Send test notification in Uptime Kuma

Note: DNS validation requires dig command. If not available, validation will be skipped (you can continue but SSL may fail).

Why This Layer is Critical:

• All infrastructure monitoring (Layer 6) depends on Uptime Kuma
• All alerts go through ntfy
• Services availability monitoring needs Uptime Kuma
• Without this layer, you won't know when things break!

---

Layer 5: VPN Infrastructure (Headscale)

Goal: Deploy Headscale for secure mesh networking (like Tailscale, but self-hosted).

Script: ./scripts/setup_layer_5_headscale.sh

This layer is OPTIONAL - Skip to Layer 6 if you don't need VPN mesh networking.

What This Layer Does:

Deploys Headscale coordination server and optionally joins machines to the mesh:

5A: Deploy Headscale Server

• Installs Headscale on spacey
• Configures with deny-all ACL policy (you customize later)
• Creates namespace/user for your network
• Sets up Caddy reverse proxy
• Configures embedded DERP server for NAT traversal
• Deployed to: spacey

5B: Join Machines to Mesh (Optional)

• Installs Tailscale client on target machines
• Generates ephemeral pre-auth keys
• Automatically joins machines to your mesh
• Enables Magic DNS
• Can join: vipy, watchtower, nodito, lapy, etc.

Prerequisites (Complete BEFORE Running):

1. Previous layers complete:

• ✅ Layer 0, 1A, 3 complete (spacey must be set up)
• ✅ Caddy running on spacey (from Layer 3)

2. Configure subdomain (in centralized config):

• ✅ Edit ansible/services_config.yml and customize headscale: under subdomains: section (e.g., headscale or vpn)

3. Create DNS record that matches your configured subdomain:

• ✅ Create A record: <headscale_subdomain>.<yourdomain> → spacey IP
• ✅ Wait for DNS propagation
• ✅ Verify with: dig <subdomain>.<yourdomain> should return spacey IP

4. Decide on namespace name:

• ✅ Choose a namespace for your network (default: counter-net)
• ✅ This is set in headscale_vars.yml as headscale_namespace

Run the Script:

source venv/bin/activate
cd /home/counterweight/personal_infra
./scripts/setup_layer_5_headscale.sh

The script will:

1. Validate DNS configuration
2. Deploy Headscale server
3. Offer to join machines to the mesh

Post-Deployment Steps:

Configure ACL Policies (Required for machines to communicate)

1. SSH into spacey: ssh counterweight@<spacey-ip>
2. Edit ACL file: sudo nano /etc/headscale/acl.json
3. Configure rules (example - allow all):

   {
     "ACLs": [
       {"action": "accept", "src": ["*"], "dst": ["*:*"]}
     ]
   }
   

4. Restart Headscale: sudo systemctl restart headscale

Default is deny-all for security - you must configure ACLs for machines to talk!

Join Additional Machines Manually

For machines not in inventory (mobile, desktop):

1. Install Tailscale client on device
2. Generate pre-auth key on spacey:

   ssh counterweight@<spacey-ip>
   sudo headscale preauthkeys create --user <namespace> --reusable
   

3. Connect using your Headscale server:

   tailscale up --login-server https://<headscale_subdomain>.<yourdomain> --authkey <key>
   

Automatic Uptime Kuma Monitor:

The playbook will automatically create a monitor in Uptime Kuma:

• ✅ Headscale - monitors https://<subdomain>/health
• Added to "services" monitor group
• Uses ntfy notification (if configured)
• Check every 60 seconds

Prerequisites: Uptime Kuma credentials must be in infra_secrets.yml (from Layer 4)

Verification:

The script will automatically verify:

• ✓ DNS records configured correctly
• ✓ Headscale installed and running
• ✓ Namespace created
• ✓ Caddy config created
• ✓ Machines joined (if selected)
• ✓ Monitor created in Uptime Kuma "services" group

List connected devices:

ssh counterweight@<spacey-ip>
sudo headscale nodes list

Why Use Headscale:

• Secure communication between all your machines
• Magic DNS - access machines by hostname
• NAT traversal - works even behind firewalls
• Self-hosted - full control of your VPN
• Mobile support - use official Tailscale apps

Backup:

Optional backup to lapy:

ansible-playbook -i inventory.ini services/headscale/setup_backup_headscale_to_lapy.yml

---

Layer 6: Infrastructure Monitoring

Goal: Deploy automated monitoring for disk usage, system health, and CPU temperature.

Script: ./scripts/setup_layer_6_infra_monitoring.sh

What This Layer Does:

Deploys monitoring scripts that report to Uptime Kuma:

6A: Disk Usage Monitoring

• Monitors disk usage on specified mount points
• Sends alerts when usage exceeds threshold (default: 80%)
• Creates Uptime Kuma push monitors automatically
• Organizes monitors in host-specific groups
• Deploys to: All hosts (selectable)

6B: System Healthcheck

• Sends regular heartbeat pings to Uptime Kuma
• Alerts if system stops responding
• "No news is good news" monitoring
• Deploys to: All hosts (selectable)

6C: CPU Temperature Monitoring (Nodito only)

• Monitors CPU temperature on Proxmox server
• Alerts when temperature exceeds threshold (default: 80°C)
• Deploys to: nodito (if configured)

Prerequisites (Complete BEFORE Running):

1. Previous layers complete:

• ✅ Layer 0, 1A/1B, 4 complete
• ✅ Uptime Kuma deployed and configured (Layer 4)
• ✅ CRITICAL: infra_secrets.yml has Uptime Kuma credentials

2. Uptime Kuma API credentials ready:

• ✅ Must have completed Layer 4 post-deployment steps
• ✅ ansible/infra_secrets.yml must contain:

  uptime_kuma_username: "your-username"
  uptime_kuma_password: "your-password"
  

3. Python dependencies installed:

• ✅ uptime-kuma-api must be in requirements.txt
• ✅ Should already be installed from Layer 0
• ✅ Verify: pip list | grep uptime-kuma-api

Run the Script:

source venv/bin/activate
cd /home/counterweight/personal_infra
./scripts/setup_layer_6_infra_monitoring.sh

The script will:

1. Verify Uptime Kuma credentials
2. Offer to deploy disk usage monitoring
3. Offer to deploy system healthchecks
4. Offer to deploy CPU temp monitoring (nodito only)
5. Test monitor creation and alerts

What Gets Deployed:

For each monitored host:

• Push monitor in Uptime Kuma (upside-down mode)
• Monitor group named {hostname} - infra
• Systemd service for monitoring script
• Systemd timer for periodic execution
• Log file for monitoring history

Default settings (customizable):

• Disk usage threshold: 80%
• Disk check interval: 15 minutes
• Healthcheck interval: 60 seconds
• CPU temp threshold: 80°C
• Monitored mount point: / (root)

Customization Options:

Change thresholds and intervals:

# Disk monitoring with custom settings
ansible-playbook -i inventory.ini infra/410_disk_usage_alerts.yml \
  -e "disk_usage_threshold_percent=85" \
  -e "disk_check_interval_minutes=10" \
  -e "monitored_mount_point=/home"

# Healthcheck with custom interval
ansible-playbook -i inventory.ini infra/420_system_healthcheck.yml \
  -e "healthcheck_interval_seconds=30"

# CPU temp with custom threshold
ansible-playbook -i inventory.ini infra/nodito/40_cpu_temp_alerts.yml \
  -e "temp_threshold_celsius=75"

Verification:

The script will automatically verify:

• ✓ Uptime Kuma API accessible
• ✓ Monitors created in Uptime Kuma
• ✓ Monitor groups created
• ✓ Systemd services running
• ✓ Can send test alerts

Check Uptime Kuma web UI:

• Monitors should appear organized by host
• Should receive test pings
• Alerts will show when thresholds exceeded

Post-Deployment:

Monitor your infrastructure:

1. Open Uptime Kuma web UI
2. See all monitors organized by host groups
3. Configure notification rules per monitor
4. Set up status pages (optional)

Test alerts:

# Trigger disk usage alert (fill disk temporarily)
# Trigger healthcheck alert (stop the service)
# Check ntfy for notifications

Why This Layer is Important:

• Proactive monitoring - Know about issues before users do
• Disk space alerts - Prevent services from failing
• System health - Detect crashed/frozen machines
• Temperature monitoring - Prevent hardware damage
• Organized - All monitors grouped by host

---

Layer 7: Core Services

Goal: Deploy core applications: Vaultwarden, Forgejo, and LNBits.

Script: ./scripts/setup_layer_7_services.sh

What This Layer Does:

Deploys main services on vipy:

7A: Vaultwarden (Password Manager)

• Deploys via Docker
• Configures Caddy reverse proxy
• Sets up fail2ban protection
• Enables sign-ups initially (disable after creating first user)
• Deployed to: vipy

7B: Forgejo (Git Server)

• Installs Forgejo binary
• Creates git user and directories
• Configures Caddy reverse proxy
• Enables SSH cloning
• Deployed to: vipy

7C: LNBits (Lightning Wallet)

• Installs system dependencies and uv (Python 3.12 tooling)
• Clones LNBits version v1.3.1
• Syncs dependencies with uv targeting Python 3.12
• Configures with FakeWallet backend (for testing)
• Creates systemd service
• Configures Caddy reverse proxy
• Deployed to: vipy

Prerequisites (Complete BEFORE Running):

1. Previous layers complete:

• ✅ Layer 0, 1A, 2, 3 complete
• ✅ Docker installed on vipy (Layer 2)
• ✅ Caddy running on vipy (Layer 3)

2. Configure subdomains (in centralized config):

• ✅ Edit ansible/services_config.yml and customize subdomains under subdomains: section:
  • Set vaultwarden: to your preferred subdomain (e.g., vault or passwords)
  • Set forgejo: to your preferred subdomain (e.g., git or code)
  • Set lnbits: to your preferred subdomain (e.g., lnbits or wallet)

3. Create DNS records matching your subdomains:

• ✅ Create A record: <vaultwarden_subdomain>.<yourdomain> → vipy IP
• ✅ Create A record: <forgejo_subdomain>.<yourdomain> → vipy IP
• ✅ Create A record: <lnbits_subdomain>.<yourdomain> → vipy IP
• ✅ Wait for DNS propagation

Run the Script:

source venv/bin/activate
cd /home/counterweight/personal_infra
./scripts/setup_layer_7_services.sh

The script will:

1. Validate DNS configuration
2. Offer to deploy each service
3. Configure backups (optional)

Post-Deployment Steps:

Vaultwarden:

1. Visit https://<vaultwarden_subdomain>.<yourdomain>
2. Create your first user account
3. Important: Disable sign-ups after first user:

   ansible-playbook -i inventory.ini services/vaultwarden/disable_vaultwarden_sign_ups_playbook.yml
   

4. Optional: Set up backup to lapy

Forgejo:

1. Visit https://<forgejo_subdomain>.<yourdomain>
2. Create admin account on first visit
3. Default: registrations disabled for security
4. SSH cloning works automatically after adding SSH key

LNBits:

1. Visit https://<lnbits_subdomain>.<yourdomain>
2. Create superuser on first visit
3. Important: Default uses FakeWallet (testing only)
4. Configure real Lightning backend:
   • Edit /opt/lnbits/lnbits/.env on vipy
   • Or use the superuser UI to configure backend
5. Disable new user registration for security
6. Optional: Set up encrypted backup to lapy

Backup Configuration:

After services are stable, set up backups:

Vaultwarden backup:

ansible-playbook -i inventory.ini services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml

LNBits backup (GPG encrypted):

ansible-playbook -i inventory.ini services/lnbits/setup_backup_lnbits_to_lapy.yml

Note: Forgejo backups are not automated - backup manually or set up your own solution.

Automatic Uptime Kuma Monitors:

The playbooks will automatically create monitors in Uptime Kuma for each service:

• ✅ Vaultwarden - monitors https://<subdomain>/alive
• ✅ Forgejo - monitors https://<subdomain>/api/healthz
• ✅ LNBits - monitors https://<subdomain>/api/v1/health

All monitors:

• Added to "services" monitor group
• Use ntfy notification (if configured)
• Check every 60 seconds
• 3 retries before alerting

Prerequisites: Uptime Kuma credentials must be in infra_secrets.yml (from Layer 4)

Verification:

The script will automatically verify:

• ✓ DNS records configured
• ✓ Services deployed
• ✓ Docker containers running (Vaultwarden)
• ✓ Systemd services running (Forgejo, LNBits)
• ✓ Caddy configs created

Manual verification:

• Visit each service's subdomain
• Create admin/first user accounts
• Test functionality
• Check Uptime Kuma for new monitors in "services" group

Why These Services:

• Vaultwarden - Self-hosted password manager (Bitwarden compatible)
• Forgejo - Self-hosted Git server (GitHub/GitLab alternative)
• LNBits - Lightning Network wallet and accounts system

---

Layer 8: Secondary Services

Status: 🔒 Locked (Complete Layer 7 first)

---

Troubleshooting

Common Issues

SSH Connection Fails

• Verify VPS is running and accessible
• Check SSH key is in the correct location
• Ensure SSH key has correct permissions (600)
• Try manual SSH: ssh -i ~/.ssh/counterganzua root@<ip>

Ansible Not Found

• Make sure you've activated the venv: source venv/bin/activate
• Run Layer 0 script again

DNS Not Resolving

• DNS changes can take up to 24-48 hours to propagate
• Use dig <subdomain>.<domain> to check DNS status
• You can proceed with setup; services will work once DNS propagates

---

Progress Tracking

Use this checklist to track your progress:

• Layer 0: Foundation Setup
• Layer 1A: VPS Basic Setup
• Layer 1B: Nodito (Proxmox) Setup
• Layer 2: General Infrastructure Tools
• Layer 3: Reverse Proxy (Caddy)
• Layer 4: Core Monitoring & Notifications
• Layer 5: VPN Infrastructure (Headscale)
• Layer 6: Infrastructure Monitoring
• Layer 7: Core Services
• Layer 8: Secondary Services
• Backups Configured