[Feature Request]: Improve dynamic splitting in BigQuery Read API's Java Connector
Open
gcp
io
java
new feature
P2
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
Trying to run large Dataflow pipelines which use the BQ Read API sporadically results in pipeline failure with the following error message:
`worker lost contact with the service`
This problem gets worse as the customer starts approaching quota limits defined in the BigQuery Read API. The connector needs to be redesigned to prevent these issues from occurring.
### Issue Priority
Priority: 2
### Issue Component
Component: io-java-gcp
Contributor guide
Assessment
This issue has not been assessed yet.