googleapis / googleapis/google-cloud-go

bigquery: allow callers to configure query job polling retries

Open
#20,121 0 comments 0 reactions 0 assignees View on GitHub
api: bigquery triage me
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

## Is your feature request related to a problem? Please describe.
`Job.waitForQuery` polls `jobs.getQueryResults` while waiting for query jobs to complete. Polling errors currently retry internally using `jobRetryReasons`, which includes `jobRateLimitExceeded` and `internalError`.

Under high query fanout, BigQuery can return `jobRateLimitExceeded` during polling. Because the client retries that internally, callers cannot observe the backpressure, log it, slow down, cancel cleanly, or fail fast. Higher-level systems can appear to hang silently until their context expires.

## Describe the solution you'd like
Add an API-compatible option that lets callers configure query job polling retry behavior without changing the default behavior.

For example:

```go
client, err := bigquery.NewClient(ctx, projectID,
bigquery.WithJobPollingRetryPolicy(bigquery.JobPollingRetryPolicy{
DisableRetries: true,
}),
)
```

The option should apply narrowly to query job polling via jobs.getQueryResults, not to job creation retries for jobs.insert or jobs.query.

Ideally it should allow callers to:

- keep the current default behavior
- disable retries for polling errors
- optionally customize polling retry reasons/backoff

## Describe alternatives you've considered
One alternative is to remove jobRateLimitExceeded from the default polling retry reasons. That would surface backpressure immediately, but it would also change behavior for existing callers.

Another alternative is to set shorter context deadlines. That does not solve the problem because the SDK still hides the real polling error until the deadline expires.

A third option is for callers to avoid Job.Read / query polling and manually poll job status. That is more work for callers and does not help code paths that rely on the existing BigQuery client abstractions.

## Additional context
Related history:

- #3795 requested adding jobRateLimitExceeded as a retryable job error.
- #9726 added jobRateLimitExceeded to jobRetryReasons.
- #10558 reduced default job polling backoff.
- #5518 is a broader BigQuery job retry improvement issue.
- #20070 proposes changing the default polling retry reason set.

Contributor guide

Open the contributing guide

Research direction

Start at Job.waitForQuery and its jobs.getQueryResults polling path, then trace how jobRetryReasons and existing client options are wired. Review related retry tests and add coverage showing that defaults remain unchanged while polling retries can be disabled or customized without affecting job creation. Done means callers can observe or configure polling errors without changing jobs.insert or jobs.query behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.