counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

a few tiny changes in readme and profiles template

Browse source
This commit is contained in:
Pablo Martin 2024-05-28 14:29:31 +02:00
parent b59c0a96fb
commit 40a12df950
2 changed files with 8 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

8
README.md
View file

@ -18,9 +18,11 @@ Welcome to Superhog's DWH dbt project. Here we model the entire DWH.
- Activate the virtual environment and run `pip install -r requirements.txt` - Activate the virtual environment and run `pip install -r requirements.txt`
- Create an entry for this project `profiles.yml` file at `~/.dbt/profiles.yml`. You have a suggested template at `profiles.yml.example` - Create an entry for this project `profiles.yml` file at `~/.dbt/profiles.yml`. You have a suggested template at `profiles.yml.example`
- Make sure that the `profiles.yml` host and port settings are consistent with whatever networking approach you've taken. - Make sure that the `profiles.yml` host and port settings are consistent with whatever networking approach you've taken.
- Run `chmod 600 ~/.dbt/profiles.yml` to secure your profiles file.
- Run `dbt deps` to install dbt dependencies
- Check - Check
- Ensure you are running in the project venv, either by setting VSCode Python interpreter to the one created by `poetry`, or by running `poetry shell` in the console when in the root dir. - Ensure you are running in the project venv, either by setting VSCode Python interpreter to the one created by `poetry`, or by running `poetry shell` in the console when in the root dir.
- Turn on your tunnel to `dev` and run `dbt debug`. If it runs well, you are all set. If it fails, there's something wrong with your set up. Grab the terminal output and pull the thread. - Run `dbt debug`. If it runs well, you are all set. If it fails, there's something wrong with your set up. Grab the terminal output and pull the thread.
- Complements - Complements
- If you are in VSCode, you most probably want to have this extension installed: [dbt Power User](https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user) - If you are in VSCode, you most probably want to have this extension installed: [dbt Power User](https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user)
- It is advised to use [this autoformatter](https://sqlfmt.com/) and to automatically [run it on save](https://docs.sqlfmt.com/integrations/vs-code). - It is advised to use [this autoformatter](https://sqlfmt.com/) and to automatically [run it on save](https://docs.sqlfmt.com/integrations/vs-code).
@ -38,9 +40,9 @@ The only missing bit to make your local deployment be like the production DWH is
For example, if you are working on models that use data from Core, you can dump and restore from your terminal with something roughly like this: For example, if you are working on models that use data from Core, you can dump and restore from your terminal with something roughly like this:
```bash ```bash
pg_dump -h superhog-dwh-prd.postgres.database.azure.com -U airbyte_user -W -F plain dwh -n sync_core > core.dump pg_dump -h superhog-dwh-prd.postgres.database.azure.com -U airbyte_user -W -F t dwh -n sync_xero_superhog_limited > xero.dump
pg_restore -h localhost -U postgres -W -d dwh core.dump pg_restore -h localhost -U postgres -W -d dwh xero.dump
``` ```
## Branching strategy ## Branching strategy

6
profiles.yml.example
View file

@ -6,7 +6,7 @@ dwh_dbt:
user: <your-user> user: <your-user>
pass: <your-password> pass: <your-password>
port: <your-port-probably-5432> port: <your-port-probably-5432>
schema: working schema: staging
threads: 4 threads: 4
type: postgres type: postgres
@ -16,8 +16,8 @@ dwh_dbt:
user: <prd-dbt-user> user: <prd-dbt-user>
pass: <prd-dbt-password> pass: <prd-dbt-password>
port: <your-port-probably-5432> port: <your-port-probably-5432>
schema: working schema: staging
threads: 4 threads: 2
type: postgres type: postgres
target: dev target: dev