personal_infra/02_vps_core_services_setup.md

# 02 VPS Core Services Setup

Now that the VPSs are ready, we need to deploy some basic services which are foundational for the apps we're actually interested in.

This assumes you've completed the markdown `01`.

## General tools

This repo contains some rather general tools that you may or may not need depending on what services you want to deploy and what device you're working on. This tools can be installed with the `900` group of playbooks sitting at `ansible/infra`. 

By default, these playbooks are configured for `hosts: all`. Be mindful if you want to limit, you can use the `--limit groupname` flag when running the playbook.

Below you have notes on adding each specific tool to a device.

### rsync

Simply run the playbook:

```
ansible-playbook -i inventory.ini infra/900_install_rsync.yml
```

### docker and compose

Simply run the playbook:

```
ansible-playbook -i inventory.ini infra/910_docker_playbook.yml
```

Checklist:
- [ ] All 3 VPSs responde to `docker version`
- [ ] All 3 VPSs responde to `docker compose version`

## Deploy Caddy

* Use Ansible to run the caddy playbook:
    
    ```
    cd ansible
    ansible-playbook -i inventory.ini services/caddy_playbook.yml
    ```

* Starting config will be empty. Modifying the caddy config file to add endpoints as we add services is covered by the instructions of each service.

Checklist:
- [ ] All 3 VPSs have Caddy up and running


## Uptime Kuma

Uptime Kuma gets used to monitor the availability of services, keep track of their uptime and notify issues.

### Deploy

* Decide what subdomain you want to serve Uptime Kuma on and add it to `services/services_config.yml` on the `uptime_kuma` entry.
    * Note that you will have to add a DNS entry to point to the VPS public IP.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/uptime_kuma/deploy_uptime_kuma_playbook.yml`.

### Set up backups to Lapy

* Make sure rsync is available on the host and on Lapy.
* Run the backup playbook: `ansible-playbook -i inventory.ini services/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml`.
* A first backup process gets executed and then a cronjob is set up to refresh backups periodically.

### Configure

* Uptime Kuma will be available for you to create a user on first start. Do that and store the creds safe.
* From that point on, you can configure through the Web UI.

### Restoring to a previous state

* Stop Uptime Kuma.
* Overwrite the data folder with one of the backups.
* Start it up again.

Checklist:
- [ ] Uptime kuma is accesible at the FQDN
- [ ] The backup script runs fine
- [ ] You have stored the credentials of the Uptime kuma admin user


## ntfy

ntfy is a notifications server.

### Deploy

* Decide what subdomain you want to serve ntfy on and add it to `services/ntfy/ntfy_vars.yml` on the `ntfy_subdomain`.
    * Note that you will have to add a DNS entry to point to the VPS public IP.
* Ensure the admin user credentials are set in `ansible/infra_secrets.yml` under `ntfy_username` and `ntfy_password`. This user is the only one authorised to send and read messages from topics.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy/deploy_ntfy_playbook.yml`.
* Run this playbook to create a notifaction entry in uptime kuma that points to your freshly deployed ntfy instance: `ansible-playbook -i inventory.ini services/ntfy/setup_ntfy_uptime_kuma_notification.yml`

### Configure

* You can visit the ntfy web UI at the FQDN you configured.
* You can start using notify to send alerts with uptime kuma by visiting the uptime kuma UI and using the credentials for the ntfy admin user.
* To receive alerts on your phone, install the official ntfy app: https://github.com/binwiederhier/ntfy-android.
* You can also subscribe on the web UI on your laptop.

### Backups

Given that ntfy is almost stateless, no backups are made. If it blows up, simply set it up again.

Checklist
- [ ] ntfy UI is reachable
- [ ] You can see the notification in uptime kuma and test it successfully

## VPS monitoring scripts

### Deploy

- Run playbooks:
    - `ansible-playbook -i inventory.ini infra/410_disk_usage_alerts.yml --limit vps`
    - `ansible-playbook -i inventory.ini infra/420_system_healthcheck.yml --limit vps`

Checklist:
- [ ] You can see both the system healthcheck and disk usage check for all VPSs in the uptime kuma UI.
- [ ] The checks are all green after ~30 min.

## Vaultwarden

Vaultwarden is a credentials manager.

### Deploy

* Decide what subdomain you want to serve Vaultwarden on and add it to `services/vaultwarden/vaultwarden_vars.yml` on the `vaultwarden_subdomain`.
    * Note that you will have to add a DNS entry to point to the VPS public IP.
* Make sure docker is available on the host.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/vaultwarden/deploy_vaultwarden_playbook.yml`.

### Configure

* Vaultwarden will be available for you to create a user on first start. Do that and store the creds safely.
* From that point on, you can configure through the Web UI.

### Disable registration

* You probably don't want anyone to just be able to register without permission.
* To prevent that, you can run the playbook `disable_vaultwarden_sign_ups_playbook.yml` after creating the first user.

### Set up backups to Lapy

* Make sure rsync is available on the host and on Lapy.
* Run the backup playbook: `ansible-playbook -i inventory.ini services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml`.
* A first backup process gets executed and then a cronjob is set up to refresh backups periodically.

### Restoring to a previous state

* Stop Vaultwarden.
* Overwrite the data folder with one of the backups.
* Start it up again.
* Be careful! The restoring of a backup doesn't include the signup behaviour. If you deployed a new instance and restored a backup, you still need to manually repeat as described above the disabling of the sign ups.

Checklist
- [ ] The service is reachable at the URL
- [ ] You have stored the admin creds properly
- [ ] You can't create another user at the /signup path

## Forgejo

Forgejo is a git server.

### Deploy

* Decide what subdomain you want to serve Forgejo on and add it to `services/forgejo/forgejo_vars.yml` on the `forgejo_subdomain`.
    * Note that you will have to add a DNS entry to point to the VPS public IP.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/forgejo/deploy_forgejo_playbook.yml`.

### Configure

* Forgejo will be available for you to create a user on first start. Do that and store the creds safely.
* Default behaviour after that is not allow for registrations.
* You can tweak more settings from that point on.
* SSH cloning should work out of the box (after you've set up your SSH pub key in Forgejo, that is).

### Set up backups to Lapy

* Make sure rsync is available on the host and on Lapy.
* Ensure GPG is configured with a recipient in your inventory (the backup script requires `gpg_recipient` to be set).
* Run the backup playbook: `ansible-playbook -i inventory.ini services/forgejo/setup_backup_forgejo_to_lapy.yml`.
* A first backup process gets executed and then a cronjob is set up to refresh backups periodically. The script backs up both the data and config directories. Backups are GPG encrypted for safety. Note that the Forgejo service is stopped during backup to ensure consistency.

### Restoring to a previous state

* Stop Forgejo.
* Decrypt the backup: `gpg --decrypt forgejo-backup-YYYY-MM-DD.tar.gz.gpg | tar -xzf -`
* Overwrite the data and config directories with the restored backup.
* Ensure that files in `/var/lib/foregejo/` are owned by the right user.
* Start Forgejo again.
* You may need to refresh ssh pub key so your old SSH driven git remotes work. Go to site administration, dashboard, and run task `Update the ".ssh/authorized_keys" file with Forgejo SSH keys.`.

Checklist:
- [ ] Forgejo is accessible at the FQDN
- [ ] You have stored the admin credentials properly
- [ ] The backup script runs fine
- [ ] SSH cloning works after setting up your SSH pub key


## Phoenixd

phoenixd is the server version of the Phoenix wallet: a self-custodial Lightning node that leans on the ACINQ node for liquidity and channel management. It is the Lightning backend that LNBits talks to, so deploy it first.

It is not exposed to the internet. Its HTTP API is bound to `127.0.0.1` and only local consumers (LNBits, running on the same host) use it. There is no subdomain and no Caddy config for it.

### Deploy

* Review `ansible/services/phoenixd/phoenixd_vars.yml`. The things worth a look:
    * `phoenixd_version`: pinned release tag.
    * `phoenixd_auto_liquidity`: how much inbound liquidity phoenixd buys when it needs some (`off`, `2m`, `5m`, `10m`). This costs sats, it is not free.
    * `phoenixd_http_bind_port`: the loopback API port LNBits will point at.
* No new secrets are needed: phoenixd generates its own API password on first start.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/phoenixd/deploy_phoenixd_playbook.yml`.
* The playbook will:
    * Install `phoenixd` and `phoenix-cli` into `/usr/local/bin`
    * Create the `phoenix` system user and a `0700` data dir at `/opt/phoenixd/.phoenix`
    * Create and start the `phoenixd` systemd service
    * Create a health check timer that runs `phoenix-cli getinfo` every minute
    * Register a push monitor named `Phoenixd` in Uptime Kuma under the `services` group

### Back up the seed

**Do this immediately after the first deploy.** The seed is the only thing that recovers the funds.

* Run the backup playbook: `ansible-playbook -i inventory.ini services/phoenixd/setup_backup_phoenixd_to_lapy.yml`.
* This gpg encrypts `seed.dat` and `phoenix.conf` to Lapy daily and keeps 14 days. It does not stop the node and it does not back up the channel database on purpose: phoenixd keeps channel state with its peer, and restoring a stale channel db to a live node can force close channels and cost you a penalty.
* Also write the 12 words down offline, once: `sudo cat /opt/phoenixd/.phoenix/seed.dat`.

### Operate

* Status: `sudo systemctl status phoenixd`, logs: `sudo journalctl -u phoenixd -f`.
* CLI: `sudo PHOENIX_DATADIR=/opt/phoenixd/.phoenix phoenix-cli getinfo` (also `getbalance`, `listchannels`, `createinvoice`). If you changed `phoenixd_http_bind_port`, add `--http-bind-port <port>` before the subcommand.
* API password, needed to wire anything up to the node: `sudo grep '^http-password=' /opt/phoenixd/.phoenix/phoenix.conf`.

### Restoring on a fresh host

* Deploy phoenixd but stop the service before it generates a new seed, or just drop the file in before the first run:
    * `sudo mkdir -p /opt/phoenixd/.phoenix`
    * `echo "your twelve words here" | sudo tee /opt/phoenixd/.phoenix/seed.dat`
    * `sudo chown -R phoenix:phoenix /opt/phoenixd/.phoenix && sudo chmod 600 /opt/phoenixd/.phoenix/seed.dat`
* Then run the deployment playbook. Never run two nodes on the same seed at once.

Checklist:
- [ ] `phoenix-cli getinfo` returns a node id
- [ ] The `Phoenixd` monitor in Uptime Kuma is green
- [ ] The seed is backed up to Lapy *and* written down offline


## LNBits

LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd node deployed above as its Lightning backend.

### Deploy

* Decide what subdomain you want to serve LNBits on and add it to `ansible/services_config.yml` under `lnbits`.
    * Note that you will have to add a DNS entry to point to the VPS public IP.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/lnbits/deploy_lnbits_playbook.yml`.

### Configure

* LNBits will be available for you to create a superuser on first start. Do that and store the creds safely.
* From that point on, you can configure through the Web UI.
* Some advice around specifics of LNbits:
    * The default setup uses a FakeWallet backend for testing. Configure a real Lightning backend as needed by modifying the `.env` file located or using the superuser UI.
    * To use the phoenixd node deployed above, set the backend to `PhoenixdWallet` with endpoint `http://127.0.0.1:9740` and the API password from `/opt/phoenixd/.phoenix/phoenix.conf`. The deployment playbook does not do this for you, since flipping the wallet backend on a live instance is not something to do behind your back.
    * For security, disable the new users registration.

### Set up backups to Lapy

* Make sure rsync is available on the host and on Lapy.
* Run the backup playbook: `ansible-playbook -i inventory.ini services/lnbits/setup_backup_lnbits_to_lapy.yml`.
* A first backup process gets executed and then a cronjob is set up to refresh backups periodically. The script backs up both the `.env` file and the sqlite database. Backups are gpg encrypted for safety.

### Restoring to a previous state

* Stop LNBits.
* Overwrite the data folder with one of the backups.
* Start it up again.


## ntfy-emergency-app

ntfy-emergency-app is a simple web application that allows trusted people to send emergency messages via ntfy notifications. Perfect for situations where you need to be alerted immediately but don't want to enable notifications on your regular messaging apps.

### Deploy

* Decide what subdomain you want to serve the emergency app on and update `ansible/services_config.yml` under `ntfy_emergency_app`.
    * Note that you will have to add a DNS entry to point to the VPS public IP.
* Configure the ntfy settings in `ntfy_emergency_app_vars.yml`:
    * `ntfy_emergency_app_topic`: The ntfy topic to send messages to (default: "emergency")
    * `ntfy_emergency_app_ui_message`: Custom message displayed in the web interface
* Ensure `infra_secrets.yml` contains `ntfy_username` and `ntfy_password` with the credentials the app should use.
* Make sure docker is available on the host.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml`.


## Headscale

Headscale is a self-hosted Tailscale control server that allows you to create your own Tailscale network.

### Deploy

* Decide what subdomain you want to serve Headscale on and add it to `services/headscale/headscale_vars.yml` on the `headscale_subdomain`.
    * Note that you will have to add a DNS entry to point to the VPS public IP.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/headscale/deploy_headscale_playbook.yml`.

### Configure

* **Network Security**: The network starts with a deny-all policy - no devices can communicate with each other until you explicitly configure ACL rules in `/etc/headscale/acl.json`.
* After deployment, the namespace specified in `services/headscale/headscale_vars.yml` is automatically created.

### Connect devices

#### Automated method (for servers reachable via SSH from lapy)

* Use the Ansible playbook to automatically join machines to the mesh:
    ```bash
    ansible-playbook -i inventory.ini infra/920_join_headscale_mesh.yml --limit <target-host>
    ```
* The playbook will:
    * Generate an ephemeral pre-auth key (expires in 1 minute) by SSHing from lapy to the headscale server
    * Install Tailscale on the target machine
    * Configure Tailscale to connect to your headscale server
    * Enable magic DNS so devices can talk to each other by hostname

#### Manual method (for mobile apps, desktop clients, etc.)

* Install Tailscale on your devices (mobile apps, desktop clients, etc.).
* Generate a pre-auth key by SSHing into your headscale server:
    ```bash
    ssh <headscale-server>
    sudo headscale preauthkeys create --user counter-net --reusable
    ```
* Instead of using the default Tailscale login, use your headscale server:
    * Server URL: `https://headscale.contrapeso.xyz` (or your configured domain)
    * Use the pre-auth key you generated above
    * Full command: `tailscale up --login-server <YOUR_HEADSCALE_URL> --authkey <YOUR_AUTH_KEY>`
* Your devices will now be part of your private Tailscale network.

### Management

* List connected devices: `headscale nodes list`
* View users: `headscale users list`
* Generate new pre-auth keys: `headscale preauthkeys create --user counter-net --reusable`
* Remove a device: `headscale nodes delete --identifier <node-id>`

## Personal Blog

Personal Blog is a static site served by Caddy's file server.

### Deploy

* Decide what subdomain you want to serve the personal blog on and add it to `ansible/services_config.yml` under `personal_blog`.
    * Note that you will have to add a DNS entry to point to the VPS public IP.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/personal-blog/deploy_personal_blog_playbook.yml`.
* The playbook will:
    * Create the web root directory at `/var/www/pablohere.contrapeso.xyz` (or your configured domain)
    * Set up Caddy configuration with file_server directive
    * Create an Uptime Kuma monitor for the site
    * Configure proper permissions for deployment

### Set up deployment alias on Lapy

* Run the deployment alias setup playbook: `ansible-playbook -i inventory.ini services/personal-blog/setup_deploy_alias_lapy.yml`.
* This creates a `deploy-personal-blog` alias in your `.bashrc` that allows you to deploy your static site from `~/pablohere/public/` to the server.
* Source your `.bashrc` or open a new terminal to use the alias: `source ~/.bashrc`
* Deploy your site by running: `deploy-personal-blog`
    * The alias copies files via scp to a temporary location, then moves them with sudo to the web root and fixes permissions.

Checklist:
- [ ] Personal blog is accessible at the FQDN
- [ ] Uptime Kuma monitor for the blog is showing as healthy
- [ ] Deployment alias is working and you can successfully deploy files