Refactor Reparallelize as an alternative Reshuffle implementation
Open
core
improvement
java
P3
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Some DoFn based IOs like JdbcIO and RedisIO rely on a different approach to Reparallelize outputs using a combination of a an empty PCollectionView to force materialization and Reshuffle.viaRandomkey to reparallelize a PCollection. This issue extracts this transform and expose it as part of the Reshuffle to avoid repeating the code for transforms (notably IOs) that produce lots of sequentially generated data where and benefit of this alternative approach to perform better reparallelization of its output.
Imported from Jira [BEAM-9748](https://issues.apache.org/jira/browse/BEAM-9748). Original Jira may contain additional context.
Reported by: iemejia.
Contributor guide
Assessment
This issue has not been assessed yet.