dbt-labs / dbt-labs/ade-bench

Agents should not be expected to edit the contents of the `dbt_packages` directory to solve a problem

Open
#55 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
123
Forks
25
PR merge metrics
No merged PRs in 30d

Description

The contents of `dbt_packages/` should not be considered user-serviceable:
- It will be overwritten on any `dbt deps` invocation,
- It is `.gitignore`d 1 so any changes made wouldn't make it off the user's computer.

Tasks like `quickbooks001` [modify dbt_packages inline](https://github.com/thedatamates/ade-bench/blob/main/tasks/quickbooks001/solution.sh#L10-L13). This might pass the test locally, but there won't be anything to check back in to git. TBH I would fail any agent which solved a problem by modifying a gitignore'd file.

If our version of the table has an incompatible data type requiring an additional cast beyond what the Fivetran package does, the correct solution would be to create a [custom override of the model](https://docs.getdbt.com/reference/resource-configs/enabled#disable-a-model-in-a-package-in-order-to-use-your-own-version-of-the-model) and disable Fivetran's version.

1 I'm sure someone somewhere is checking their whole dbt_packages directory into git - they have strayed from god's light

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.