[Feature Request]: BigQuery Storage API read throttle
Open
gcp
io
java
new feature
P2
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
Beam version: `2.36.0`
Jobs reading from the BigQuery Storage API can hit Quota limits and eventually fail with `The worker lost contact with the service.` (Dataflow runner)
The reason for the failure can be diagnosed on the API Monitoring (GCP) for BigQuery Storage API calls, where a spike in `429`s can be seen for `ReadRows` when the throughput of the job starts going downhill.
On the GCP side we've tried to resolve it through Quota increases, but it would be helpful to de-risk jobs crashing
related to:
* https://github.com/apache/beam/pull/15445 <- Shouldn't this PR have pervented this kind of issue?
### Issue Priority
Priority: 2
### Issue Component
Component: io-java-gcp
Contributor guide
Assessment
This issue has not been assessed yet.