data-dwh-dbt-project/ci

CI

You can setup CI pipelines for the project if you want. This enables performing certain checks in PRs and master commits, which is useful to minimize errors and ensure certain quality levels are met.

The details here are specific to Azure Devops. If you need to set things up in a different Git/CI env, you'll have to adjust your way into it.

CI VM Setup

Requirements

These instructions assume that:

• You have a VM ready to be setup as the CI server.
• You can SSH into it.
• The VM has Docker and Docker Compose installed and ready to run.
• The VM has psql installed.
• The VM has the Azure CI agent installed.
• That you have cloned this repository in the home folder of the user you use in that VM.
• The DWH production instance has a CI dedicated user that can read from all sync schemas as well as staging, intermediate and reporting, and you have the credentials.

Setting things up

• Create a folder in the user home directory named dbt-ci.
• Create a copy of the ci.env file there naming it .env (cp ci.env ~/dbt-ci/.env) and fill it with values of your choice.
• Execute the script named ci-vm-setup.sh in this folder. This script will take care of most of the setup that need to be executed, including:
  • Preparing the postgres database.
  • Setting up the dockerized postgres with the right database, FDW, etc.
  • Prepare the profiles.yml file.

Connecting to Devops

• TBD