dbt-labs / dbt-labs/dbt

[Feature] Include column-level information in dbt's adapter cache, for faster get_columns_in_relation

Open
#9,595 4 comments 1 reaction 0 assignees View on GitHub
engine:v1 performance type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

I have some macros which rely heavily on get_columns_in_relation. they are macros which operate on the column level and there are a lot of calls to the same relation to get information about columns. in the end there are thousands of calls to get_columns_in_relation while there are actually 10-100 different relations.
as the calls are quite expensive it would be great to have a shared memory cache for each macro which survives the call of the macro.

it should be something like a key/value-store which would allow to add the return value of get_columns_in_relation for a specific relation and in the next macro call we would check if the value is already there and only call get_columns_in_relation if we don't have the value already.

### Describe alternatives you've considered

surely we can call get_columns_in_relation in other places and let the result flow through the codebase. but this would lead to a lot of complexity and is not a good tradeoff for the performance gain.

### Who will this benefit?

everyone who loves a performant dbt

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.