counterweight/sh-notion
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

pages

Browse source
This commit is contained in:
Pablo Martin 2025-07-11 16:15:17 +02:00
parent 729d6d6db4
commit a256b48b01
630 changed files with 16494 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

174
notion_data_team_no_files/dbt 1 7 to 1 9 upgrade 1740446ff9c98054915fd620df86339a.md Normal file
View file

@ -0,0 +1,174 @@
# dbt 1.7 to 1.9 upgrade
On Jan ‘25, we set ourselves to upgrade our dbt project version. This page tracks the task.
## Starting details
On commit `04a10cf9c52ad849ef6f61b133e605efc813e33d`, we hold the following versions in our `requirements.txt` file:
```sql
dbt-core~=1.7.6
dbt-postgres~=1.7.6
```
Furthermore, in the Airbyte production machine, we have these versions installed in the `venv` dedicated to dbt:
```sql
dbt-core==1.7.9
dbt-postgres==1.7.9
```
## Goal
To bump versions into the highest `1.9` patch, ensure everything works, provide instructions for all analysts and also inform on new features available.
At the time of writing this, the highest `1.9` patch for dbt is `1.9.1` (https://github.com/dbt-labs/dbt-core/releases/tag/v1.9.1)
As for the Postgres adapter, the most recent version is `1.9.0`.
## Steps
- [x] Backup `pip freeze` output of production dbt deployment.
- Output here
```python
agate==1.7.1
annotated-types==0.6.0
attrs==23.2.0
Babel==2.14.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
dbt-core==1.7.9
dbt-extractor==0.5.1
dbt-postgres==1.7.9
dbt-semantic-interfaces==0.4.4
idna==3.6
importlib-metadata==6.11.0
isodate==0.6.1
Jinja2==3.1.3
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
leather==0.4.0
Logbook==1.5.3
MarkupSafe==2.1.5
mashumaro==3.12
minimal-snowplow-tracker==0.0.2
more-itertools==10.2.0
msgpack==1.0.8
networkx==3.2.1
packaging==23.2
parsedatetime==2.6
pathspec==0.11.2
protobuf==4.25.3
psycopg2-binary==2.9.9
pycparser==2.21
pydantic==2.6.3
pydantic_core==2.16.3
python-dateutil==2.9.0.post0
python-slugify==8.0.4
pytimeparse==1.1.8
pytz==2024.1
PyYAML==6.0.1
referencing==0.33.0
requests==2.31.0
rpds-py==0.18.0
six==1.16.0
sqlparse==0.4.4
text-unidecode==1.3
typing_extensions==4.10.0
urllib3==1.26.18
zipp==3.17.0
```
- [x] Upgrade package versions in production dbt deployment.
- New pip freeze here.
```python
agate==1.7.1
annotated-types==0.6.0
attrs==23.2.0
Babel==2.14.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
daff==1.3.46
dbt-adapters==1.13.0
dbt-common==1.14.0
dbt-core==1.9.1
dbt-extractor==0.5.1
dbt-postgres==1.9.0
dbt-semantic-interfaces==0.7.4
deepdiff==7.0.1
idna==3.6
importlib-metadata==6.11.0
isodate==0.6.1
Jinja2==3.1.3
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
leather==0.4.0
Logbook==1.5.3
MarkupSafe==2.1.5
mashumaro==3.12
minimal-snowplow-tracker==0.0.2
more-itertools==10.2.0
msgpack==1.0.8
networkx==3.2.1
ordered-set==4.1.0
packaging==23.2
parsedatetime==2.6
pathspec==0.11.2
protobuf==5.29.2
psycopg2-binary==2.9.9
pycparser==2.21
pydantic==2.6.3
pydantic_core==2.16.3
python-dateutil==2.9.0.post0
python-slugify==8.0.4
pytimeparse==1.1.8
pytz==2024.1
PyYAML==6.0.1
referencing==0.33.0
requests==2.31.0
rpds-py==0.18.0
six==1.16.0
snowplow-tracker==1.0.4
sqlparse==0.5.3
text-unidecode==1.3
types-requests==2.32.0.20241016
typing_extensions==4.10.0
urllib3==2.3.0
zipp==3.17.0
```
- [x] Attempt to run our usual dbt run. Check if everything works and logs look good.
- [x] If shit hits the fan, rollback, study issues and go back to step 1. Do not continue down this list.
- Shit did hit the fan
- We started to get this error when running any dbt cli command: `ModuleNotFoundError: No module named 'dbt.adapters.factory'`
- We fixed it by applying this good gentleman’s advice: https://github.com/dbt-labs/dbt-core/issues/10135#issuecomment-2113728550
- [x] If all is well, open PR to bump versions in git repo.
- PR here: https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/3970
- [x] Create instructions for team to upgrade their local environments and make sure to communicate thoroughly, ask everyone to ACK back once done.
- Instructions below in this page.
- [x] Make TLDR on cool features we have obtained and reference to docs for further detail.
## Instructions for analysts
Team, we’ve upgraded our version of `dbt` to 1.9. This is already applied in our production deployment, and [this PR](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/3970) is ready to apply it on the project level.
We also need you to apply this version upgrade in your laptops so that versions are in sync across environments and stuff fits nicely. It’s very simple, you can find below the steps:
- Open your VSCode workspace for the dbt project.
- Open up a terminal and make sure it has the project virtual environment activated.
- Make a backup of your python packages in case things go wrong: `pip freeze > my_packages_backup.txt`
- Run the following sequence of commands to get things installed:
- `pip uninstall -y dbt-adapters`
- `pip install dbt-core==1.9.1 --upgrade`
- `pip install dbt-postgres==1.9.0 --upgrade`
- To check that stuff works, just try to use dbt. You can begin with a humble `dbt --version`, which should show the new version that is installed. If that works fine, move into using dbt as usual in your local env.
And that’s it! Welcome to dbt 1.9.