counterweight/sh-notion
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
sh-notion/notion_data_team_no_files/dbt 1 9 1 to 1 9 8 upgrade 2100446ff9c980cbaa01e84c22bdd13c.md

238 lines
7.6 KiB
Markdown
Raw Normal View History

pages
2025-07-11 16:15:17 +02:00
# dbt 1.9.1 to 1.9.8 upgrade
On June ‘25, we set ourselves to upgrade our dbt project version. This page tracks the task.
## Starting details
On commit `04a10cf9c52ad849ef6f61b133e605efc813e33d`, we hold the following versions in our `requirements.txt` file:
```bash
dbt-core~=1.9.1
dbt-postgres~=1.9.0
```
Furthermore, in the Airbyte production machine, we have these versions installed in the `venv` dedicated to dbt:
```bash
dbt-core==1.9.1
dbt-postgres==1.9.0
```
Besides the contents of the `requirements.txt` file, this is the contents of the full output of `pip freeze` on the Airbyte machine’s dbt environment:
```bash
agate==1.7.1
annotated-types==0.6.0
attrs==23.2.0
Babel==2.14.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
daff==1.3.46
dbt-adapters==1.13.0
dbt-common==1.14.0
dbt-core==1.9.1
dbt-extractor==0.5.1
dbt-postgres==1.9.0
dbt-semantic-interfaces==0.7.4
deepdiff==7.0.1
idna==3.6
importlib-metadata==6.11.0
isodate==0.6.1
Jinja2==3.1.3
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
leather==0.4.0
Logbook==1.5.3
MarkupSafe==2.1.5
mashumaro==3.12
minimal-snowplow-tracker==0.0.2
more-itertools==10.2.0
msgpack==1.0.8
networkx==3.2.1
ordered-set==4.1.0
packaging==23.2
parsedatetime==2.6
pathspec==0.11.2
protobuf==5.29.2
psycopg2-binary==2.9.9
pycparser==2.21
pydantic==2.6.3
pydantic_core==2.16.3
python-dateutil==2.9.0.post0
python-slugify==8.0.4
pytimeparse==1.1.8
pytz==2024.1
PyYAML==6.0.1
referencing==0.33.0
requests==2.31.0
rpds-py==0.18.0
six==1.16.0
snowplow-tracker==1.0.4
sqlparse==0.5.3
text-unidecode==1.3
types-requests==2.32.0.20241016
typing_extensions==4.10.0
urllib3==2.3.0
zipp==3.17.0
```
## Goal
To bump versions into `1.9.8` patch, ensure everything works, provide instructions for all analysts and also inform on new features available. Here’s the release link in Github: https://github.com/dbt-labs/dbt-core/releases/tag/v1.9.8
As for the Postgres adapter, the most recent version is still `1.9.0`, so there’s no need to upgrade anything.
## Steps
- [x] Backup `pip freeze` output of production dbt deployment.
- Output here
```python
agate==1.7.1
annotated-types==0.6.0
attrs==23.2.0
Babel==2.14.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
daff==1.3.46
dbt-adapters==1.13.0
dbt-common==1.14.0
dbt-core==1.9.1
dbt-extractor==0.5.1
dbt-postgres==1.9.0
dbt-semantic-interfaces==0.7.4
deepdiff==7.0.1
idna==3.6
importlib-metadata==6.11.0
isodate==0.6.1
Jinja2==3.1.3
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
leather==0.4.0
Logbook==1.5.3
MarkupSafe==2.1.5
mashumaro==3.12
minimal-snowplow-tracker==0.0.2
more-itertools==10.2.0
msgpack==1.0.8
networkx==3.2.1
ordered-set==4.1.0
packaging==23.2
parsedatetime==2.6
pathspec==0.11.2
protobuf==5.29.2
psycopg2-binary==2.9.9
pycparser==2.21
pydantic==2.6.3
pydantic_core==2.16.3
python-dateutil==2.9.0.post0
python-slugify==8.0.4
pytimeparse==1.1.8
pytz==2024.1
PyYAML==6.0.1
referencing==0.33.0
requests==2.31.0
rpds-py==0.18.0
six==1.16.0
snowplow-tracker==1.0.4
sqlparse==0.5.3
text-unidecode==1.3
types-requests==2.32.0.20241016
typing_extensions==4.10.0
urllib3==2.3.0
zipp==3.17.0
```
- [x] Upgrade package versions in production dbt deployment with `pip install dbt-core==1.9.8 --upgrade`
- New pip freeze here.
```python
agate==1.7.1
annotated-types==0.6.0
attrs==23.2.0
Babel==2.14.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
daff==1.3.46
dbt-adapters==1.13.0
dbt-common==1.14.0
dbt-core==1.9.8
dbt-extractor==0.5.1
dbt-postgres==1.9.0
dbt-semantic-interfaces==0.7.4
deepdiff==7.0.1
idna==3.6
importlib-metadata==6.11.0
isodate==0.6.1
Jinja2==3.1.3
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
leather==0.4.0
Logbook==1.5.3
MarkupSafe==2.1.5
mashumaro==3.12
minimal-snowplow-tracker==0.0.2
more-itertools==10.2.0
msgpack==1.0.8
networkx==3.2.1
ordered-set==4.1.0
packaging==23.2
parsedatetime==2.6
pathspec==0.11.2
protobuf==5.29.2
psycopg2-binary==2.9.9
pycparser==2.21
pydantic==2.6.3
pydantic_core==2.16.3
python-dateutil==2.9.0.post0
python-slugify==8.0.4
pytimeparse==1.1.8
pytz==2024.1
PyYAML==6.0.1
referencing==0.33.0
requests==2.31.0
rpds-py==0.18.0
six==1.16.0
snowplow-tracker==1.0.4
sqlparse==0.5.3
text-unidecode==1.3
types-requests==2.32.0.20241016
typing_extensions==4.10.0
urllib3==2.3.0
zipp==3.17.0
```
- [x] Attempt to run our usual dbt run. Check if everything works and logs look good.
- [x] If all is well, open PR to bump versions in git repo.
- PR here: https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/5455
- [x] Create instructions for team to upgrade their local environments and make sure to communicate thoroughly, ask everyone to ACK back once done.
- Instructions below in this page.
- [x] Make TLDR on cool features we have obtained and reference to docs for further detail.
## Instructions for analysts
Team, we’ve upgraded our version of `dbt` to `1.9.8`. This is already applied in our production deployment, and [this PR](https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/5455) is ready to apply it on the project level.
We also need you to apply this version upgrade in your laptops so that versions are in sync across environments and stuff fits nicely. It’s very simple, you can find below the steps:
- [ ] Open your VSCode workspace for the dbt project.
- [ ] Open up a terminal and make sure it has the project virtual environment activated.
- [ ] Make a backup of your python packages list in case things go wrong: `pip freeze > my_packages_backup.txt`
- [ ] Run the following commands to get things installed `pip install dbt-core==1.9.8 --upgrade`
- [ ] To check that stuff works, just try to use dbt. You can begin with a humble `dbt --version`, which should show the new version that is installed. If that works fine, move into using dbt as usual in your local env.
Regarding new stuff after this update:
- This upgrade doesn’t come with new features. Since we only moved patch versions, we only get bug fixes and performance improvements.
- Having said that, this upgrade should fix the funny “dbt breaks if I make a new version of a model but I don’t delete the `target` folder “ bug that [caused this incident](20240913-01%20-%20dbt%20run%20blocked%20by%20%E2%80%9Cnot%20in%20the%20graph%201030446ff9c980c291f1d57751f443ee.md). So you should be able to not have to ever care again about deleting the `target` folder, and we can remove the systematic deletion that we had in our production scripts (I’ll take care of verifying the absence of the error and adjusting the scripts, no need to worry about that).
And that’s it! Welcome to dbt `1.9.8`.
Reference in a new issue Copy permalink