counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

remove old docs from readme

Browse source
This commit is contained in:
Pablo Martin 2024-10-01 10:09:11 +02:00
parent f955dc4f4c
commit 3a649db5f2
1 changed files with 0 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

16
README.md
View file

@ -35,22 +35,6 @@ Running a local version of the DWH allows you to test things as you develop: a m
You can read on how to set this up in `dev-env/local_dwh.md`.
Having a database where you can run your WIP models is very useful to ease development. But obviously, we can't do that in production. We could do it in a shared dev instance, but then we would step into each others toes when developing.
To overcome these issues, we rely on local clones of the DWH. The idea is to have a PostgreSQL instance running on your laptop. You perform your `dbt run` statements for testing and you validate the outcome of your work there. When you are confident and have tested properly, you can PR to master.
You will find a docker compose file named `dev-dwh.docker-compose.yml`. It will simply start a PostgreSQL 16 database in your device. You can raise it, adjust it to your needs, and adapt the `profiles.yml` file to point to it when you are developing locally.
The only missing bit to make your local deployment be like the production DWH is to have the source data from the source systems. The current policy is to generate a dump from the production database with what you need and restore it in your local postgres. That way, you are using accurate and representative data to do your work.
For example, if you are working on models that use data from Core, you can dump and restore from your terminal with something roughly like this:
```bash
pg_dump -h superhog-dwh-prd.postgres.database.azure.com -U airbyte_user -W -F t dwh -n sync_xero_superhog_limited > xero.dump
pg_restore -h localhost -U postgres -W -d dwh xero.dump
```
## Branching strategy
This repo works in a trunk-based-development philosophy (<https://trunkbaseddevelopment.com/>).