googleapis / googleapis/google-cloud-python

feat: add `job_retry` argument to `load_table_from_uri`

Aperta
#15,738 5 commenti 0 reazioni 1 assegnatario Rivendicata da @tswast Vedi su GitHub
api: bigquery type: feature request
Lingua principale
Python
Stelle
5.4k
Fork
1.8k
Merge medio
3g 4h
PR unite (30g)
122

Descrizione

In internal issue 195911158, a customer is struggling to retry jobs that fail with "403 Exceeded rate limits: too many table update operations for this table". One can encounter this exception by attempting to run hundreds of load jobs in parallel.

Thoughts:

1. Try to reproduce. Does the exception happen at `result()` or `load_table_from_uri()`? If `result()`, continue with `job_retry`, otherwise see if we can modify the default retry predicate for `load_table_from_uri()` to find this rate limiting reason and retry.
2. Assuming the exception does happen at `result()`, modify load jobs (or more likely the base class) to retry if job retry is set, similar to what we do for query jobs.

Notes:

* I suspect we'll need a different default `job_retry` object for `load_table_from_uri()`, as the retryable reasons will likely be different than what we have for queries.
* I don't think the other `load_table_from_*` are as retryable as `load_table_from_uri()`, since they would require rewinding file objects, which isn't always possible. We'll probably want to consider adding `job_retry` to those load job methods in the future, but for now `load_table_from_uri` is what's needed.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.