sh-notion/notion_data_team_no_files/dbt 1 7 to 1 9 upgrade 1740446ff9c98054915fd620df86339a.md

dbt 1.7 to 1.9 upgrade

On Jan ‘25, we set ourselves to upgrade our dbt project version. This page tracks the task.

Starting details

On commit 04a10cf9c52ad849ef6f61b133e605efc813e33d, we hold the following versions in our requirements.txt file:

dbt-core~=1.7.6
dbt-postgres~=1.7.6

Furthermore, in the Airbyte production machine, we have these versions installed in the venv dedicated to dbt:

dbt-core==1.7.9
dbt-postgres==1.7.9

Goal

To bump versions into the highest 1.9 patch, ensure everything works, provide instructions for all analysts and also inform on new features available.

At the time of writing this, the highest 1.9 patch for dbt is 1.9.1 (https://github.com/dbt-labs/dbt-core/releases/tag/v1.9.1)

As for the Postgres adapter, the most recent version is 1.9.0.

Steps

• Backup pip freeze output of production dbt deployment.

  • Output here

    agate==1.7.1
    annotated-types==0.6.0
    attrs==23.2.0
    Babel==2.14.0
    certifi==2024.2.2
    cffi==1.16.0
    charset-normalizer==3.3.2
    click==8.1.7
    colorama==0.4.6
    dbt-core==1.7.9
    dbt-extractor==0.5.1
    dbt-postgres==1.7.9
    dbt-semantic-interfaces==0.4.4
    idna==3.6
    importlib-metadata==6.11.0
    isodate==0.6.1
    Jinja2==3.1.3
    jsonschema==4.21.1
    jsonschema-specifications==2023.12.1
    leather==0.4.0
    Logbook==1.5.3
    MarkupSafe==2.1.5
    mashumaro==3.12
    minimal-snowplow-tracker==0.0.2
    more-itertools==10.2.0
    msgpack==1.0.8
    networkx==3.2.1
    packaging==23.2
    parsedatetime==2.6
    pathspec==0.11.2
    protobuf==4.25.3
    psycopg2-binary==2.9.9
    pycparser==2.21
    pydantic==2.6.3
    pydantic_core==2.16.3
    python-dateutil==2.9.0.post0
    python-slugify==8.0.4
    pytimeparse==1.1.8
    pytz==2024.1
    PyYAML==6.0.1
    referencing==0.33.0
    requests==2.31.0
    rpds-py==0.18.0
    six==1.16.0
    sqlparse==0.4.4
    text-unidecode==1.3
    typing_extensions==4.10.0
    urllib3==1.26.18
    zipp==3.17.0
    

• Upgrade package versions in production dbt deployment.

  • New pip freeze here.

    agate==1.7.1
    annotated-types==0.6.0
    attrs==23.2.0
    Babel==2.14.0
    certifi==2024.2.2
    cffi==1.16.0
    charset-normalizer==3.3.2
    click==8.1.7
    colorama==0.4.6
    daff==1.3.46
    dbt-adapters==1.13.0
    dbt-common==1.14.0
    dbt-core==1.9.1
    dbt-extractor==0.5.1
    dbt-postgres==1.9.0
    dbt-semantic-interfaces==0.7.4
    deepdiff==7.0.1
    idna==3.6
    importlib-metadata==6.11.0
    isodate==0.6.1
    Jinja2==3.1.3
    jsonschema==4.21.1
    jsonschema-specifications==2023.12.1
    leather==0.4.0
    Logbook==1.5.3
    MarkupSafe==2.1.5
    mashumaro==3.12
    minimal-snowplow-tracker==0.0.2
    more-itertools==10.2.0
    msgpack==1.0.8
    networkx==3.2.1
    ordered-set==4.1.0
    packaging==23.2
    parsedatetime==2.6
    pathspec==0.11.2
    protobuf==5.29.2
    psycopg2-binary==2.9.9
    pycparser==2.21
    pydantic==2.6.3
    pydantic_core==2.16.3
    python-dateutil==2.9.0.post0
    python-slugify==8.0.4
    pytimeparse==1.1.8
    pytz==2024.1
    PyYAML==6.0.1
    referencing==0.33.0
    requests==2.31.0
    rpds-py==0.18.0
    six==1.16.0
    snowplow-tracker==1.0.4
    sqlparse==0.5.3
    text-unidecode==1.3
    types-requests==2.32.0.20241016
    typing_extensions==4.10.0
    urllib3==2.3.0
    zipp==3.17.0
    

• Attempt to run our usual dbt run. Check if everything works and logs look good.

  • If shit hits the fan, rollback, study issues and go back to step 1. Do not continue down this list.
  • Shit did hit the fan
    • We started to get this error when running any dbt cli command: ModuleNotFoundError: No module named 'dbt.adapters.factory'
    • We fixed it by applying this good gentleman’s advice: https://github.com/dbt-labs/dbt-core/issues/10135#issuecomment-2113728550

• If all is well, open PR to bump versions in git repo.

  • PR here: https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/3970

• Create instructions for team to upgrade their local environments and make sure to communicate thoroughly, ask everyone to ACK back once done.

  • Instructions below in this page.

• Make TLDR on cool features we have obtained and reference to docs for further detail.

Instructions for analysts

Team, we’ve upgraded our version of dbt to 1.9. This is already applied in our production deployment, and this PR is ready to apply it on the project level.

We also need you to apply this version upgrade in your laptops so that versions are in sync across environments and stuff fits nicely. It’s very simple, you can find below the steps:

• Open your VSCode workspace for the dbt project.
• Open up a terminal and make sure it has the project virtual environment activated.
• Make a backup of your python packages in case things go wrong: pip freeze > my_packages_backup.txt
• Run the following sequence of commands to get things installed:
  • pip uninstall -y dbt-adapters
  • pip install dbt-core==1.9.1 --upgrade
  • pip install dbt-postgres==1.9.0 --upgrade
• To check that stuff works, just try to use dbt. You can begin with a humble dbt --version, which should show the new version that is installed. If that works fine, move into using dbt as usual in your local env.

And that’s it! Welcome to dbt 1.9.