googleapis / googleapis/google-cloud-python

feat: add `job_retry` argument to `load_table_from_uri`

Offen
#15,738 5 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @tswast Auf GitHub ansehen
api: bigquery type: feature request
Vorherrschende Sprache
Python
Sterne
5.4k
Forks
1.8k
Ø Merge
3 T. 4 Std.
Gemergte PRs (30 T.)
122

Beschreibung

In internal issue 195911158, a customer is struggling to retry jobs that fail with "403 Exceeded rate limits: too many table update operations for this table". One can encounter this exception by attempting to run hundreds of load jobs in parallel.

Thoughts:

1. Try to reproduce. Does the exception happen at `result()` or `load_table_from_uri()`? If `result()`, continue with `job_retry`, otherwise see if we can modify the default retry predicate for `load_table_from_uri()` to find this rate limiting reason and retry.
2. Assuming the exception does happen at `result()`, modify load jobs (or more likely the base class) to retry if job retry is set, similar to what we do for query jobs.

Notes:

* I suspect we'll need a different default `job_retry` object for `load_table_from_uri()`, as the retryable reasons will likely be different than what we have for queries.
* I don't think the other `load_table_from_*` are as retryable as `load_table_from_uri()`, since they would require rewinding file objects, which isn't always possible. We'll probably want to consider adding `job_retry` to those load job methods in the future, but for now `load_table_from_uri` is what's needed.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.