googleapis / googleapis/google-cloud-python

perf: don't retry getQueryResults as often with ambiguous errors in `QueryJob.result()`

Aperta
#15,714 5 commenti 0 reazioni 1 assegnatario Rivendicata da @tswast Vedi su GitHub
api: bigquery bigframes priority: p3
Lingua principale
Python
Stelle
5.4k
Fork
1.8k
Merge medio
3g 4h
PR unite (30g)
122

Descrizione

> [!IMPORTANT]
> Do not change the retries for `jobs.getQueryResults` REST API calls in `RowIterator`, only in `QueryJob.result()` where the HTTP status code are ambiguous. Once we get to `RowIterator` we know the job has succeeded and these errors cease to be ambigous.

**Is your feature request related to a problem? Please describe.**

When a job fails due to quota issues, the [`jobs.getQueryResults` BigQuery REST API](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/getQueryResults) translates job failure status into failure HTTP codes. This means that exceptions like `google.api_core.exceptions.TooManyRequests` are ambiguous. It could be at the Google Frontend level and mean we've hit our API request quota or could be because the job has failed due to query quota issues.

Because these ambiguous errors are included in the default retry predicate, we end up retrying the `jobs.getQueryResults` request until our retries expire. Only after which do we do a call to `jobs.get` to see that the job has failed for a retriable reason.

**Describe the solution you'd like**

Likely we need the default retry object for our calls to `jobs.getQueryResults` (the ones where we know we're waiting for a job to complete, _not_ the ones where we're downloading the results) to be different from the default retry for all other API requests.

This may require an additional parameter to `QueryJob.result()` and the methods that call it.

**Describe alternatives you've considered**

Note: This issue has been mitigated by https://github.com/googleapis/python-bigquery/pull/1734 which ensures that the default `job_retry` has a deadline that exceeds the default deadline of `retry`. It means we don't retry the job nearly as quickly as we could, though.

**Additional context**

See the discussion at https://github.com/googleapis/python-bigquery/pull/1900/files#r1565837480 also internal folks can see similar discussions on issue 311358887 and 312216177 in the Java client.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.