Add support for dynamic worker re-balancing when reading BigQuery data using Cloud Dataflow
Open
gcp
io
java
new feature
P3
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently, the BigQuery connector for reading data using the BigQuery Storage API does not support any of the facilities on the source for Dataflow to split streams.
On the server side, the BigQuery Storage API supports splitting streams at a fraction. By adding support to the connector, we enable Dataflow to split streams, which unlocks dynamic worker re-balancing.
Imported from Jira [BEAM-7495](https://issues.apache.org/jira/browse/BEAM-7495). Original Jira may contain additional context.
Reported by: aryann.
Contributor guide
Assessment
This issue has not been assessed yet.