20240913-01 - dbt run blocked by “not in the graph” error

`dbt run` blocked by “not in the graph” error

Managed by: Pablo

Summary

Components involved: DWH, dbt
Started at: When did the issue actually start
Detected at: When did we notice that the incident existed
Mitigated at: When did we bring things to a stable state without further impact

We deployed for the first time a version of our dbt project that used the versioning features of dbt. An active bug on dbt core prevented any dbt commands like dbt run or dbt test to work because the compilation of the project would fail. The issue was resolved by applying a somewhat patchy workaround that enables dbt to work again properly.

Impact

None beyond some noise in the alerts channel and making Pablo’s Friday afternoon hectic.

Timeline

Keeping it simple on this one since there isn’t much value in tracking stuff in hyper detail.

All reported times are in CEST Timezone.

Time	Event
2024-09-13 15:24	Pablo merges PR #2771 in the dbt project
2024-09-13 15:25	Pablo manually triggers the `run_dbt.sh` script in production and the execution fails
2024-09-13 15:25-15:52	Pablo scrambles around trying to understand what the heck is happening.
2024-09-13 15:52	Pablo manages to get a first successful `dbt run` after applying one of the workarounds suggested by dbt labs
	End of the incident.

Root Cause(s)

The root cause is a bug in dbt when it attempts to parse and compile the project. The bug is triggered by adding a new version to an existing model that didn’t have versions before. It is unknown at this point if this bug is also triggered when adding an additional version to a model that is already being versioned. The bug is identified by dbt labs and is sitting in this issue in the dbt core repository: https://github.com/dbt-labs/dbt-core/issues/8872

Within our platform, the issue was introduced by the following PR in our dbt project repository: https://guardhog.visualstudio.com/Data/_git/data-dwh-dbt-project/pullrequest/2771. The PR introduced a new version for model int_core__verification_payments, which triggered the dbt core bug when we started to run dbt commands in production.

Resolution and recovery

I manually modified our deployed dbt_run.sh script in production to include a dbt clean and dbt deps commands before we executed the run ones. This deleted the target folder and fixed the issue, since this is one of the suggested workarounds. Subsequent executions of the script after this fix ran perfectly fine.

Lessons Learned

What went well:

What went badly

This issue already happened in my local (Pablo) some days before, but I dismissed it as some silly flaky behaviour. I guess I probably randomly executed one of the workarounds (running a dbt clean) and in the process, accidentally fixed the issue without really understanding what had happened. The lesson here is to not dismiss quirky behaviours in dbt and to try to understand them fully (even reproduce them if necessary) so that we can be confident and in control at all times.

Where did we get lucky:

The fact that the issue was already spotted and documented in the official dbt core repository made handling the situation much simpler. Had there been no public showcase of the bug source and workarounds, we would have had a bad time fixing and understanding stuff since it would have required to dive into the internals of dbt.

Besides these lessons, I would also suggest this was a great reminder of the fact that the open source tools we rely on are by no means perfect, and that we must be alert when stuff goes south and always consider the option that they have bugs.

Action Items

Judge our options around Blue/Green deployments, which would enable issues like this to happen without making a single scratch in the DWH where consumers are reading from (besides an inevitable delay in the refreshing of data).
Track the dbt bug (https://github.com/dbt-labs/dbt-core/issues/8872) so that we can adjust our code once it’s fixed

4.2 KiB Raw Permalink Blame History Unescape Escape