personal_infra/ansible/roles/forgejo_runner/README.md

# `forgejo_runner`

Installs and runs a Forgejo Actions runner, registers it with the Forgejo
instance, and keeps a health check on a systemd timer.

Converted from `deploy_forgejo_runner_playbook.yml` (409 lines) under Plan 6.
The playbook is now 16 lines.

## Phases

`tasks/main.yml` imports five files in order:

| | |
|---|---|
| `prerequisites.yml` | Docker must be present |
| `install.yml` | binary, system user, working directory |
| `configure.yml` | config file, registration with the instance |
| `service.yml` | systemd unit, start, assert it came up |
| `healthcheck.yml` | check script, unit, timer |

`import_tasks`, not `include_tasks` — static imports are visible to
`--list-tasks`, which is how the conversion was verified against the playbook it
replaced.

## Monitoring: one variable, no product knowledge

This role contains **nothing specific to any monitoring system**. What used to
be here — an ~80-line embedded Python script creating monitors over the Uptime
Kuma API, a `/tmp` credentials file, token extraction, a systemd `Environment=`
rewrite, and 8 `when: uptime_kuma_enabled` guards — is gone.

What remains answers the actual question, *is this service healthy*, and records
it two ways:

- **the exit code**, which systemd keeps: `systemctl is-failed
  forgejo-runner-healthcheck.service` is a complete answer with no monitoring
  system involved at all;
- **a log file** at `{{ healthcheck_log_file }}`.

To report health somewhere, set one variable:

```yaml
healthcheck_push_url: "https://example/api/push/TOKEN"
```

Any endpoint accepting an HTTP ping works. Empty (the default) means check, log,
exit honestly, report nowhere — which is also the right setting for a *pull*-based
monitor like Prometheus' textfile collector, since that reads unit state instead.

The push URL is a credential (anyone holding it can forge an "up"), so callers
pass it from the vault rather than committing it.

## One behaviour change, deliberate

`Assert runner is running` used to be guarded by `uptime_kuma_enabled`, so it
never ran. It is not a monitoring task — it is the deployment checking its own
work — and the deprecation banner swept it up by mistake. It is ungated here,
which means a runner that fails to start now fails the play instead of
deploying "successfully" in silence.

## Docker-in-Docker for jobs

`forgejo_runner_dind_enabled: true` gives Actions jobs a Docker daemon of their
own — a `docker:dind` container on this host's daemon, isolated from it — so
`docker build` and `docker compose` work inside a workflow.

It follows Forgejo's own [Utilizing Docker within
Actions](https://forgejo.org/docs/latest/admin/actions/docker-access/) →
Docker-in-Docker, which is **three** parts, not one:

| | what it does | where it applies |
|---|---|---|
| `forgejo-dind.service` | a `docker:dind` container, published on `127.0.0.1:2376` only | the host |
| `container.docker_host` | `tcp://127.0.0.1:2376` — the runner creates job/service/step containers **on dind** | the runner process |
| `runner.envs.DOCKER_HOST` + `container.options` | `tcp://forgejo-dind.docker.internal:2375`, resolved by `--add-host=…:host-gateway` | inside each job container |

The third row is the one that is easy to miss, and leaving it out fails in a
way that looks identical to not having configured anything. `docker_host` only
decides **where the job container is created**; for a `tcp://` value Forgejo
mounts nothing into that container, so the job still has no daemon. The two
addresses differ because they are used from different places — the runner is a
process on the host and reaches dind on loopback; a job container is *inside*
dind, where `127.0.0.1` is itself. `host-gateway` is expanded by the daemon
that creates the container, which is dind, so it lands on dind's own bridge
gateway where its `dockerd` is listening.

### What changes for jobs

- Job containers, service containers and anything a job starts all live on the
  dind daemon. A `postgres` service and a `docker compose` stack are siblings
  there, so they reach each other by name — which is what makes end-to-end
  tests work.
- **Nothing a workflow does can touch this host's Docker daemon any more.**
  That is the point of dind over `automount`.
- dind has its own image store. Every image is pulled again the first time it
  is needed. It is a named volume (`forgejo-dind-lib`), so it survives the
  container being recreated, and `forgejo-dind-prune.timer` drops weekly what
  has gone a week unused.
- Bind mounts inside a job's own `docker compose` resolve on the **dind**
  filesystem, not the job container's. A compose file that bind-mounts the
  checked-out repo will not see it. Named volumes and `build:` contexts are
  fine. This is inherent to dind, not to this role.

### Things to know before enabling

- **This runner is shared.** `arbret`, `arbret-biz`, `arbretdos` and
  `arbret-ops` build here too, and `container.docker_host` is per *runner*, not
  per repo — every one of them moves onto dind. Their workflows do not change;
  their next run pulls images into a cold cache.
- **`capacity` is 4 and dind is one daemon.** Concurrent jobs can see and touch
  each other's containers and images through the Docker CLI. Forgejo's docs
  call this out. Fine for a single-tenant runner; set `capacity: 1` if
  untrusted code ever runs here.
- **The dind container is `--privileged`**, which is why this belongs on a
  dedicated CI VM and not beside anything that matters. `forgejo-runner-box`
  runs the runner and nothing else.
- **The daemon has no TLS and no authentication.** It is published on
  `127.0.0.1` only, so reaching it means already having a shell on this box —
  but it does mean any local user is root in a privileged container. Do not
  move `forgejo_runner_dind_bind_address` off loopback.
- **The `runner` user is still in the host `docker` group**, left over from
  when the runner used the host daemon. With dind it no longer needs it. Worth
  removing, deliberately, as its own change.
- **`cache.host` is autodetected**, and job containers now reach the runner's
  cache server across dind's NAT. Autodetect returns this box's LAN address,
  which is routable from in there. If `actions/cache` starts silently missing,
  pin `forgejo_runner_cache_host` — that is the first place to look.
- **Deploy when the runner is idle.** `shutdown_timeout` is 3h, so restarting
  it mid-job waits for the job.

### Turning it off

Set `forgejo_runner_dind_enabled: false` and re-run. The role stops and
disables the units, removes them, and puts `container.docker_host` back to
`"-"` — the exact configuration this host ran before. The image and the
`forgejo-dind-lib` volume are left alone: they are a cache, they hold no
secrets, and deleting them on a toggle is not the role's call.

## `config.yml` is templated now

It used to be `forgejo-runner generate-config`, run once on first deploy and
never touched again — so the repo had no say in anything inside it, including
the setting this feature needs. It is now `templates/config.yml.j2`.

Every value in it was transcribed from the file live on `forgejo-runner-box`
and diffed key-by-key against it: with dind disabled, 30 of 30 keys are
identical. Two deliberate differences:

- `runner.envs` no longer carries `A_TEST_ENV_NAME_1` / `A_TEST_ENV_NAME_2`,
  upstream's example placeholders, which were being injected into every job.
- the upstream comment blob is gone; the values are not.

Because the file is managed, a change to it now restarts the runner
(`handlers/main.yml`). Previously a config change could not happen at all; the
failure mode where it lands on disk and the running process never picks it up
is one this repo has hit before.