Enable the ability to grow partition count in the underlying Spark RDD
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
When using a `HadoopInputFormatIO` the number of splits seems to be controlled by the underlying `InputFormat` which in turn determines the number of partitions and therefore parallelisation when running on Spark. It is possible to `Reshuffle` the data to compensate for data skew, but it _appears_ there is no way to grow the number of partitions. The `GroupCombineFunctions.reshuffle` seems to be the only place calling the Spark `repartition` and it uses the number of partitions from the original RDD.
Scenarios that would benefit from this:
- Increasing parallelisation for computationally heavy stages
- ETLs where the input partitions are dictated by the source while you wish to optimise the partitions for fast loading to the target sink
- Zip files (my case) where they are read in single threaded manner with a custom HadoopInputFormat and therefore get a single task for all stages
(It would be nice if a user could supply a partitioner too, to help dictate data locality)
Imported from Jira [BEAM-3022](https://issues.apache.org/jira/browse/BEAM-3022). Original Jira may contain additional context.
Reported by: timrobertson100.
Contributor guide
Research direction
Start by reading the HadoopInputFormatIO path and GroupCombineFunctions.reshuffle, where the issue says Spark repartition currently uses the original RDD partition count. Define and implement the requested way for users to grow partition counts, then verify that computational stages and sink-loading stages can use the increased parallelism.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100