counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

some changes in networking instructions

Browse source
This commit is contained in:
Pablo Martin 2024-02-01 15:32:50 +01:00
parent 4e430a94a9
commit dbcfb9d075
1 changed files with 4 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

9
README.md
View file

@ -8,12 +8,12 @@ Welcome to Superhog's DWH dbt project. Here we model the entire DWH.
- You need a Linux environment. That can be Linux, macOS or WSL. - You need a Linux environment. That can be Linux, macOS or WSL.
- You need to install Python `>=3.10` and `poetry`. - You need to install Python `>=3.10` and `poetry`.
- All docs will assume you are using VSCode. - All docs will assume you are using VSCode.
- Prepare SSH tunnels - Prepare networking
- We currently use SSH tunnels to reach both the `dev` and `prd` instances. You can ask Pablo how to set these up. - You must be able to reach the DWH server through the network. There are several ways to do this.
- You will need to activate the tunnels in order to run the dbt models on the databases. It will probably pay off to make them easy to activate in your terminal, you can make an alias. - The current recommended route is to use the data VPN. You can ask Pablo to help you set it up.
- Set up - Set up
- Create an entry for this project `profiles.yml` file at `~/.dbt/profiles.yml`. You have a suggested template at `profiles.yml.example` - Create an entry for this project `profiles.yml` file at `~/.dbt/profiles.yml`. You have a suggested template at `profiles.yml.example`
- Make sure that the `profiles.yml` host and port settings are consistent with the tunnels. - Make sure that the `profiles.yml` host and port settings are consistent with whatever networking approach you've taken.
- Use `poetry install` to get dependencies in place. - Use `poetry install` to get dependencies in place.
- Check - Check
- Ensure you are running in the project venv, either by setting VSCode Python interpreter to the one created by `poetry`, or by running `poetry shell` in the console when in the root dir. - Ensure you are running in the project venv, either by setting VSCode Python interpreter to the one created by `poetry`, or by running `poetry shell` in the console when in the root dir.
@ -63,5 +63,4 @@ We organize models in four folders:
- Automate formatting with git pre-commit. - Automate formatting with git pre-commit.
- Define conventions on testing (and enforce them). - Define conventions on testing (and enforce them).
- Define conventions on documentation (and enforce them). - Define conventions on documentation (and enforce them).
- Replace SSH tunneling with a Wireguard VPN access.
- Prepare a quick way to replicate parts of the `prd` dwh in our local machines. - Prepare a quick way to replicate parts of the `prd` dwh in our local machines.