counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

stuff

Browse source
This commit is contained in:
counterweight 2025-11-14 23:36:00 +01:00
parent c8754e1bdc
commit fbbeb59c0e
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
28 changed files with 907 additions and 995 deletions
Download patch file Download diff file Expand all files Collapse all files

140
scripts/README.md
View file

@ -1,140 +0,0 @@
# Infrastructure Setup Scripts
This directory contains automated setup scripts for each layer of the infrastructure.
## Overview
Each script handles a complete layer of the infrastructure setup:
- Prompts for required variables
- Validates prerequisites
- Creates configuration files
- Executes playbooks
- Verifies completion
## Usage
Run scripts in order, completing one layer before moving to the next:
### Layer 0: Foundation Setup
```bash
./scripts/setup_layer_0.sh
```
Sets up Ansible control node on your laptop.
### Layer 1A: VPS Basic Setup
```bash
source venv/bin/activate
./scripts/setup_layer_1a_vps.sh
```
Configures users, SSH, firewall, and fail2ban on VPS machines (vipy, watchtower, spacey).
**Runs independently** - no Nodito required.
### Layer 1B: Nodito (Proxmox) Setup
```bash
source venv/bin/activate
./scripts/setup_layer_1b_nodito.sh
```
Configures Nodito Proxmox server: bootstrap, community repos, optional ZFS.
**Runs independently** - no VPS required.
### Layer 2: General Infrastructure Tools
```bash
source venv/bin/activate
./scripts/setup_layer_2.sh
```
Installs rsync and docker on hosts that need them.
- **rsync:** For backup operations (vipy, watchtower, lapy recommended)
- **docker:** For containerized services (vipy, watchtower recommended)
- Interactive: Choose which hosts get which tools
### Layer 3: Reverse Proxy (Caddy)
```bash
source venv/bin/activate
./scripts/setup_layer_3_caddy.sh
```
Deploys Caddy reverse proxy on VPS machines (vipy, watchtower, spacey).
- **Critical:** All web services depend on Caddy
- Automatic HTTPS with Let's Encrypt
- Opens firewall ports 80/443
- Creates sites-enabled directory structure
### Layer 4: Core Monitoring & Notifications
```bash
source venv/bin/activate
./scripts/setup_layer_4_monitoring.sh
```
Deploys ntfy and Uptime Kuma on watchtower.
- **ntfy:** Notification service for alerts
- **Uptime Kuma:** Monitoring platform for all services
- **Critical:** All infrastructure monitoring depends on these
- Sets up backups (optional)
- **Post-deploy:** Create Uptime Kuma admin user and update infra_secrets.yml
### Layer 5: VPN Infrastructure (Headscale)
```bash
source venv/bin/activate
./scripts/setup_layer_5_headscale.sh
```
Deploys Headscale VPN mesh networking on spacey.
- **OPTIONAL** - Skip to Layer 6 if you don't need VPN
- Secure mesh networking between all machines
- Magic DNS for hostname resolution
- NAT traversal support
- Can join machines automatically or manually
- Post-deploy: Configure ACL policies for machine communication
### Layer 6: Infrastructure Monitoring
```bash
source venv/bin/activate
./scripts/setup_layer_6_infra_monitoring.sh
```
Deploys automated monitoring for infrastructure.
- **Requires:** Uptime Kuma credentials in infra_secrets.yml (Layer 4)
- Disk usage monitoring with auto-created push monitors
- System healthcheck (heartbeat) monitoring
- CPU temperature monitoring (nodito only)
- Interactive selection of which hosts to monitor
- All monitors organized by host groups
### Layer 7: Core Services
```bash
source venv/bin/activate
./scripts/setup_layer_7_services.sh
```
Deploys core services on vipy: Vaultwarden, Forgejo, LNBits.
- Password manager (Vaultwarden) with /alive endpoint
- Git server (Forgejo) with /api/healthz endpoint
- Lightning wallet (LNBits) with /api/v1/health endpoint
- **Automatic:** Creates Uptime Kuma monitors in "services" group
- **Requires:** Uptime Kuma credentials in infra_secrets.yml
- Optional: Configure backups to lapy
### Layer 8+
More scripts will be added as we build out each layer.
## Important Notes
1. **Centralized Configuration:**
- All service subdomains are configured in `ansible/services_config.yml`
- Edit this ONE file instead of multiple vars files
- Created automatically in Layer 0
- DNS records must match the subdomains you configure
2. **Always activate the venv first** (except for Layer 0):
```bash
source venv/bin/activate
```
3. **Complete each layer fully** before moving to the next
4. **Scripts are idempotent** - safe to run multiple times
5. **Review changes** before confirming actions
## Getting Started
1. Read `../human_script.md` for the complete guide
2. Start with Layer 0
3. Follow the prompts
4. Proceed layer by layer

32
scripts/setup_layer_6_infra_monitoring.sh
View file

@ -374,44 +374,16 @@ deploy_cpu_temp_monitoring() {
echo " • Check interval: 60 seconds"
echo ""
# Check if nodito_secrets.yml exists
if [ ! -f "$ANSIBLE_DIR/infra/nodito/nodito_secrets.yml" ]; then
print_warning "nodito_secrets.yml not found"
print_info "You need to create this file with Uptime Kuma push URL"
if confirm_action "Create nodito_secrets.yml now?"; then
# Get Uptime Kuma URL
local root_domain=$(grep "^root_domain:" "$ANSIBLE_DIR/infra_vars.yml" | awk '{print $2}' 2>/dev/null)
local uk_subdomain=$(grep "^uptime_kuma_subdomain:" "$ANSIBLE_DIR/services/uptime_kuma/uptime_kuma_vars.yml" | awk '{print $2}' 2>/dev/null || echo "uptime")
echo -e -n "${BLUE}Enter Uptime Kuma push URL${NC} (e.g., https://${uk_subdomain}.${root_domain}/api/push/xxxxx): "
read push_url
mkdir -p "$ANSIBLE_DIR/infra/nodito"
cat > "$ANSIBLE_DIR/infra/nodito/nodito_secrets.yml" << EOF
# Nodito Secrets
# DO NOT commit to git
# Uptime Kuma Push URL for CPU temperature monitoring
nodito_uptime_kuma_cpu_temp_push_url: "${push_url}"
EOF
print_success "Created nodito_secrets.yml"
else
print_warning "Skipping CPU temp monitoring"
return 0
fi
fi
echo ""
if ! confirm_action "Proceed with CPU temp monitoring deployment?"; then
print_warning "Skipped"
return 0
fi
print_info "Running: ansible-playbook -i inventory.ini infra/nodito/40_cpu_temp_alerts.yml"
print_info "Running: ansible-playbook -i inventory.ini infra/430_cpu_temp_alerts.yml"
echo ""
if ansible-playbook -i inventory.ini infra/nodito/40_cpu_temp_alerts.yml; then
if ansible-playbook -i inventory.ini infra/430_cpu_temp_alerts.yml; then
print_success "CPU temperature monitoring deployed"
return 0
else

363
scripts/setup_layer_8_secondary_services.sh Executable file
View file

@ -0,0 +1,363 @@
#!/bin/bash
###############################################################################
# Layer 8: Secondary Services
#
# This script deploys the ntfy-emergency-app and memos services.
# Must be run after Layers 0-7 are complete.
###############################################################################
set -e # Exit on error
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Project directories
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
ANSIBLE_DIR="$PROJECT_ROOT/ansible"
declare -a LAYER_SUMMARY=()
print_header() {
echo -e "\n${BLUE}========================================${NC}"
echo -e "${BLUE}$1${NC}"
echo -e "${BLUE}========================================${NC}\n"
}
print_success() {
echo -e "${GREEN}✓${NC} $1"
}
print_error() {
echo -e "${RED}✗${NC} $1"
}
print_warning() {
echo -e "${YELLOW}⚠${NC} $1"
}
print_info() {
echo -e "${BLUE}ℹ${NC} $1"
}
confirm_action() {
local prompt="$1"
local response
read -p "$(echo -e ${YELLOW}${prompt}${NC} [y/N]: )" response
[[ "$response" =~ ^[Yy]$ ]]
}
record_summary() {
LAYER_SUMMARY+=("$1")
}
get_hosts_from_inventory() {
local group="$1"
cd "$ANSIBLE_DIR"
ansible-inventory -i inventory.ini --list | \
python3 -c "import sys, json; data=json.load(sys.stdin); print(' '.join(data.get('$group', {}).get('hosts', [])))" 2>/dev/null || echo ""
}
get_primary_host_ip() {
local group="$1"
cd "$ANSIBLE_DIR"
ansible-inventory -i inventory.ini --list | \
python3 -c "import sys, json; data=json.load(sys.stdin); hosts=data.get('$group', {}).get('hosts', []); print(hosts[0] if hosts else '')" 2>/dev/null || echo ""
}
check_prerequisites() {
print_header "Verifying Prerequisites"
local errors=0
if [ -z "$VIRTUAL_ENV" ]; then
print_error "Virtual environment not activated"
echo "Run: source venv/bin/activate"
((errors++))
else
print_success "Virtual environment activated"
fi
if ! command -v ansible &> /dev/null; then
print_error "Ansible not found"
((errors++))
else
print_success "Ansible found"
fi
if [ ! -f "$ANSIBLE_DIR/inventory.ini" ]; then
print_error "inventory.ini not found"
((errors++))
else
print_success "inventory.ini exists"
fi
if [ ! -f "$ANSIBLE_DIR/infra_vars.yml" ]; then
print_error "infra_vars.yml not found"
((errors++))
else
print_success "infra_vars.yml exists"
fi
if [ ! -f "$ANSIBLE_DIR/services_config.yml" ]; then
print_error "services_config.yml not found"
((errors++))
else
print_success "services_config.yml exists"
fi
if ! grep -q "^\[vipy\]" "$ANSIBLE_DIR/inventory.ini"; then
print_error "vipy not configured in inventory.ini"
((errors++))
else
print_success "vipy configured in inventory"
fi
if ! grep -q "^\[memos-box\]" "$ANSIBLE_DIR/inventory.ini"; then
print_warning "memos-box not configured in inventory.ini (memos deployment will be skipped)"
else
print_success "memos-box configured in inventory"
fi
if [ $errors -gt 0 ]; then
print_error "Prerequisites not met. Resolve the issues above and re-run the script."
exit 1
fi
print_success "Prerequisites verified"
# Display configured subdomains
local emergency_subdomain=$(grep "^ ntfy_emergency_app:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "emergency")
local memos_subdomain=$(grep "^ memos:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "memos")
print_info "Configured subdomains:"
echo " • ntfy_emergency_app: $emergency_subdomain"
echo " • memos: $memos_subdomain"
echo ""
}
check_dns_configuration() {
print_header "Validating DNS Configuration"
if ! command -v dig &> /dev/null; then
print_warning "dig command not found. Skipping DNS validation."
print_info "Install dnsutils/bind-tools to enable DNS validation."
return 0
fi
cd "$ANSIBLE_DIR"
local root_domain
root_domain=$(grep "^root_domain:" "$ANSIBLE_DIR/infra_vars.yml" | awk '{print $2}' 2>/dev/null)
if [ -z "$root_domain" ]; then
print_error "Could not determine root_domain from infra_vars.yml"
return 1
fi
local emergency_subdomain=$(grep "^ ntfy_emergency_app:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "emergency")
local memos_subdomain=$(grep "^ memos:" "$ANSIBLE_DIR/services_config.yml" | awk '{print $2}' 2>/dev/null || echo "memos")
local vipy_ip
vipy_ip=$(get_primary_host_ip "vipy")
if [ -z "$vipy_ip" ]; then
print_error "Unable to determine vipy IP from inventory"
return 1
fi
local memos_ip=""
if grep -q "^\[memos-box\]" "$ANSIBLE_DIR/inventory.ini"; then
memos_ip=$(get_primary_host_ip "memos-box")
fi
local dns_ok=true
local emergency_fqdn="${emergency_subdomain}.${root_domain}"
local memos_fqdn="${memos_subdomain}.${root_domain}"
print_info "Expected DNS:"
echo " • $emergency_fqdn → $vipy_ip"
if [ -n "$memos_ip" ]; then
echo " • $memos_fqdn → $memos_ip"
else
echo " • $memos_fqdn → (skipped - memos-box not in inventory)"
fi
echo ""
local resolved
print_info "Checking $emergency_fqdn..."
resolved=$(dig +short "$emergency_fqdn" | head -n1)
if [ "$resolved" = "$vipy_ip" ]; then
print_success "$emergency_fqdn resolves to $resolved"
elif [ -n "$resolved" ]; then
print_error "$emergency_fqdn resolves to $resolved (expected $vipy_ip)"
dns_ok=false
else
print_error "$emergency_fqdn does not resolve"
dns_ok=false
fi
if [ -n "$memos_ip" ]; then
print_info "Checking $memos_fqdn..."
resolved=$(dig +short "$memos_fqdn" | head -n1)
if [ "$resolved" = "$memos_ip" ]; then
print_success "$memos_fqdn resolves to $resolved"
elif [ -n "$resolved" ]; then
print_error "$memos_fqdn resolves to $resolved (expected $memos_ip)"
dns_ok=false
else
print_error "$memos_fqdn does not resolve"
dns_ok=false
fi
fi
echo ""
if [ "$dns_ok" = false ]; then
print_error "DNS validation failed."
print_info "Update DNS records as shown above and wait for propagation."
echo ""
if ! confirm_action "Continue anyway? (SSL certificates will fail without correct DNS)"; then
exit 1
fi
else
print_success "DNS validation passed"
fi
}
deploy_ntfy_emergency_app() {
print_header "Deploying ntfy-emergency-app"
cd "$ANSIBLE_DIR"
print_info "This deploys the emergency notification interface pointing at ntfy."
echo ""
if ! confirm_action "Deploy / update the ntfy-emergency-app?"; then
print_warning "Skipped ntfy-emergency-app deployment"
record_summary "${YELLOW}• ntfy-emergency-app${NC}: skipped"
return 0
fi
print_info "Running: ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml"
echo ""
if ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml; then
print_success "ntfy-emergency-app deployed successfully"
record_summary "${GREEN}• ntfy-emergency-app${NC}: deployed"
else
print_error "ntfy-emergency-app deployment failed"
record_summary "${RED}• ntfy-emergency-app${NC}: failed"
fi
}
deploy_memos() {
print_header "Deploying Memos"
if ! grep -q "^\[memos-box\]" "$ANSIBLE_DIR/inventory.ini"; then
print_warning "memos-box not in inventory. Skipping memos deployment."
record_summary "${YELLOW}• memos${NC}: skipped (memos-box missing)"
return 0
fi
cd "$ANSIBLE_DIR"
if ! confirm_action "Deploy / update memos on memos-box?"; then
print_warning "Skipped memos deployment"
record_summary "${YELLOW}• memos${NC}: skipped"
return 0
fi
print_info "Running: ansible-playbook -i inventory.ini services/memos/deploy_memos_playbook.yml"
echo ""
if ansible-playbook -i inventory.ini services/memos/deploy_memos_playbook.yml; then
print_success "Memos deployed successfully"
record_summary "${GREEN}• memos${NC}: deployed"
else
print_error "Memos deployment failed"
record_summary "${RED}• memos${NC}: failed"
fi
}
verify_services() {
print_header "Verifying Deployments"
cd "$ANSIBLE_DIR"
local ssh_key=$(grep "ansible_ssh_private_key_file" "$ANSIBLE_DIR/inventory.ini" | head -n1 | sed 's/.*ansible_ssh_private_key_file=\([^ ]*\).*/\1/')
ssh_key="${ssh_key/#\~/$HOME}"
local vipy_host
vipy_host=$(get_hosts_from_inventory "vipy")
if [ -n "$vipy_host" ]; then
print_info "Checking services on vipy ($vipy_host)..."
if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$vipy_host "docker ps | grep ntfy-emergency-app" &>/dev/null; then
print_success "ntfy-emergency-app container running"
else
print_warning "ntfy-emergency-app container not running"
fi
echo ""
fi
if grep -q "^\[memos-box\]" "$ANSIBLE_DIR/inventory.ini"; then
local memos_host
memos_host=$(get_hosts_from_inventory "memos-box")
if [ -n "$memos_host" ]; then
print_info "Checking memos on memos-box ($memos_host)..."
if timeout 5 ssh -i "$ssh_key" -o StrictHostKeyChecking=no -o BatchMode=yes counterweight@$memos_host "systemctl is-active memos" &>/dev/null; then
print_success "memos systemd service running"
else
print_warning "memos systemd service not running"
fi
echo ""
fi
fi
}
print_summary() {
print_header "Layer 8 Summary"
if [ ${#LAYER_SUMMARY[@]} -eq 0 ]; then
print_info "No actions were performed."
return
fi
for entry in "${LAYER_SUMMARY[@]}"; do
echo -e "$entry"
done
echo ""
print_info "Next steps:"
echo " • Visit each service's subdomain to complete any manual setup."
echo " • Configure backups for new services if applicable."
echo " • Update Uptime Kuma monitors if additional endpoints are desired."
}
main() {
print_header "Layer 8: Secondary Services"
check_prerequisites
check_dns_configuration
deploy_ntfy_emergency_app
deploy_memos
verify_services
print_summary
}
main "$@"