# Concourse Note

On 2025-10-07, I wanted to modify CI to build a new docker image in CI out of `lana-bank`. 

I teamed up with Justin who gave me a nice overview of what's happening in `lana-bank`'s concourse CI. Was good both for learning about concourse and `lana-bank`'s specifics.

There's a lot to unpack, so I'm writing some notes before all the knowledge flies away:

- Hierarchy
    - Pipelines (such as `lana-bank`) contain...
        - Groups (such as `lana-bank` and `nix-cache`) are dumb namespaces to group...
            - Jobs (the colored boxes in the UI, which get run)

- Each job gets defined through one single YAML file. This YAML file gets generated dynamically from the scripts under `ci` named `repipe`. This `repipe` scripts use `ytt`, a templating tool. `repipe`:
    - Composes the final job output
    - Applies it to the pipeline to define what runs on concourse
    - Note: you can actually "test in production" by running the `repipe` script while your local `fly` CLI is pointing to our concourse production instance. It will modify the actual production job definition there.

- Jobs use resources, which are external states
- State can either be an input (`get`) or an output (`put`). `get`s just get fetched, `put`s get mutated.
    - Resources have a `type`, which specifies what it really means to `get` or `put` them. There are many included `type`s in concourse, but you can also build your own custom ones if needed.


- Some stuff on `ytt`
  - The special characters to reference values are `#@`.
  - You can create a file to drop values to keep it all tidy and then reference it in the target file (we set values in `values.yml`, then reference them in `pipeline.yml`)
  - `ytt` not only provides simple values templating but also more sophisticated python function passing.


- On resources in a job:
  - On `get` resources, specifying `trigger: true` defines that any updates to that resources should trigger a job run.
  - Also on `get` resources, specifying `passed` while pointing to other jobs will signal that a certain resource should only be fetched if a certain job has built successfully. This chains jobs and prevents running downstream if upstream is failing.
  - Even if you define resources in `get` and `put` in a job, you still need to define them again `inputs` and `outputs` within the `task` entry so they are available (technically mounted).
  - This is because the `get`, `task` and `put` parts are all runnable steps. Calling `get` runs it, but doesn't make the state available to `task` by default. That's why you need `inputs`.


- On secrets
  - Many of the values that we interpolate with `ytt` are actually references to our Hashicorp `Vault`. They can be spotted because they use double parantheses `(( some_secret ))`. This get replaced at runtime in concourse. It's fine to hardcode stuff in `values.yml` in the repo, but secrets must go into the `Vault`.