dbt-labs / dbt-labs/dbt-adapters

[Feature] Make timeout for bigquery temp table dynamic.

Open
#805 0 comments 0 reactions 0 assignees View on GitHub
pkg:dbt-bigquery type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

**Current State**
At the moment temp tables created in big query dbt models have arbitrary timeout set to 12h.

**Code Reference**
https://github.com/dbt-labs/dbt-adapters/blob/999dd7fd8238259538e271938cb5d1fc7e65c335/dbt-bigquery/src/dbt/adapters/bigquery/impl.py#L777

**Idea for the feature**
It would be useful to customize this timeout and align it with actual timeout for processing.

**Idea for follow up**
Once we customize timeout - it will give us foundation to also customize temp table name and make it per run - at the moment it's always `_temp` which will prevent e.g. concurrent runs targeting different partitions with `insert_overwrite` per partition strategy.
I think in order to get there though - it's essential to align timeouts.

### Describe alternatives you've considered

_No response_

### Who will this benefit?

People who run the same model many times, with high failures rate (since if query fails temp table won't be dropped).

### Are you interested in contributing this feature?

Yup, I'll add PR shortly + I signed CLA

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.