data-dwh-dbt-project/ci/README.md

# CI

You can setup CI pipelines for the project if you want. This enables performing certain checks in PRs and master commits, which is useful to minimize errors and ensure certain quality levels are met.

The details here are specific to Azure Devops. If you need to set things up in a different Git/CI env, you'll have to adjust your way into it.

## CI VM Setup

### Requirements

These instructions assume that:
- You have a VM ready to be setup as the CI server.
- You can SSH into it.
- The VM has Docker and Docker Compose installed and ready to run.
- The VM has `psql` installed.
- The VM has the Azure CI agent installed.
- That you have cloned this repository in the home folder of the user you use in that VM.
- The DWH production instance has a CI dedicated user that can read from all sync schemas as well as `staging`, `intermediate` and `reporting`, and you have the credentials.

If you don't have this, it probably means you need to review our Infrastructure repository where we describe how to set a VM up with all of this.

### Setting things up

- SSH into the CI VM.
- Create a folder in the user home directory named `dbt-ci`.
- Create a copy of the `ci.env` file there naming it `.env` (`cp ci.env ~/dbt-ci/.env`) and fill it with values of your choice.
- Execute the script named `ci-vm-setup.sh` in this folder. This script will take care of most of the setup that need to be executed, including:
    - Preparing the postgres database.
    - Setting up the dockerized postgres with the right database, FDW, etc.
    - Prepare the `profiles.yml` file.

### Testing

- If the infra was set correctly and you followed the previous steps, you should be ready to roll.
- You might want to activate pipeline executions in Devops if you had it off while preparing everything.
- Once that's done:
  - Create a branch in this repository.
  - Add some silly change to any dbt model.
  - Open a PR in Devops from the branch.
- If everything is fine, you should see in Devops the pipeline getting triggered automatically and walking through all the steps described in `.azure-pipelines.master.yml`.
- Once you make a commit to `master` or merge PR to `master`, you should also see pipelines getting triggered automatically `.azure-pipelines.master.yml`.
start ci folder 2025-03-20 12:24:32 +01:00			`# CI`

wip 2025-04-02 15:00:22 +02:00			`You can setup CI pipelines for the project if you want. This enables performing certain checks in PRs and master commits, which is useful to minimize errors and ensure certain quality levels are met.`

			`The details here are specific to Azure Devops. If you need to set things up in a different Git/CI env, you'll have to adjust your way into it.`

			`## CI VM Setup`

			`### Requirements`

			`These instructions assume that:`
			`- You have a VM ready to be setup as the CI server.`
			`- You can SSH into it.`
			`- The VM has Docker and Docker Compose installed and ready to run.`
			- The VM has `psql` installed.
			`- The VM has the Azure CI agent installed.`
			`- That you have cloned this repository in the home folder of the user you use in that VM.`
			- The DWH production instance has a CI dedicated user that can read from all sync schemas as well as `staging`, `intermediate` and `reporting`, and you have the credentials.

more stuff in readme 2025-04-02 15:54:41 +02:00			`If you don't have this, it probably means you need to review our Infrastructure repository where we describe how to set a VM up with all of this.`

wip 2025-04-02 15:00:22 +02:00			`### Setting things up`

more stuff in readme 2025-04-02 15:54:41 +02:00			`- SSH into the CI VM.`
wip 2025-04-02 15:00:22 +02:00			- Create a folder in the user home directory named `dbt-ci`.
			- Create a copy of the `ci.env` file there naming it `.env` (`cp ci.env ~/dbt-ci/.env`) and fill it with values of your choice.
			- Execute the script named `ci-vm-setup.sh` in this folder. This script will take care of most of the setup that need to be executed, including:
			`- Preparing the postgres database.`
			`- Setting up the dockerized postgres with the right database, FDW, etc.`
			- Prepare the `profiles.yml` file.

more stuff in readme 2025-04-02 15:54:41 +02:00			`### Testing`
wip 2025-04-02 15:00:22 +02:00
more stuff in readme 2025-04-02 15:54:41 +02:00			`- If the infra was set correctly and you followed the previous steps, you should be ready to roll.`
			`- You might want to activate pipeline executions in Devops if you had it off while preparing everything.`
			`- Once that's done:`
			`- Create a branch in this repository.`
			`- Add some silly change to any dbt model.`
			`- Open a PR in Devops from the branch.`
			- If everything is fine, you should see in Devops the pipeline getting triggered automatically and walking through all the steps described in `.azure-pipelines.master.yml`.
			- Once you make a commit to `master` or merge PR to `master`, you should also see pipelines getting triggered automatically `.azure-pipelines.master.yml`.