dbt-labs / dbt-labs/dbt-adapters

[Feature] update dbt-bigquery retry options

Open
#864 0 comments 0 reactions 0 assignees View on GitHub
pkg:dbt-bigquery type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Expose 2 new parameters for the bigquery retry functionality - give google some more time to spin up or find resources:

1. base_retry_delay
2. exponential_sleep_multiplier

https://cloud.google.com/storage/docs/retry-strategy
(I did not find a similar page specific to BQ)

Bigquery has issues with jobs and resources not being available.
About 1/1000 bq jobs randomly seems to fail. Retrying the job solves most of those issues.
But , sometimes google just needs some more time.
Configure a delay and stop the random job failures.

### Describe alternatives you've considered

Manually restarting the job an hour later normally solves everything.

### Who will this benefit?

Any user of dbt-bigquery with a large n of models.

"The Business". Stakeholders don't seem to care that google had an intermittent issue. SLA's are brutal.

### Are you interested in contributing this feature?

Sure!

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.