Add dynamic splitting to JdbcIO.readWithPartitions
Open
improvement
io
java
jdbc
P2
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Now, the JDBC IO is basically a `DoFn` executed with a `{}ParDo{`}. So, it means that parallelism is "limited" and executed on one executor. ReadWithPartitions does some preliminary partitioning of the data, but any skew in data range or workload will create an unbalanced workload.
Imported from Jira [BEAM-14161](https://issues.apache.org/jira/browse/BEAM-14161). Original Jira may contain additional context.
Reported by: pabloem.
Contributor guide
Assessment
This issue has not been assessed yet.