apache / apache/beam

Enable the ability to grow partition count in the underlying Spark RDD

Open
#18,550 0 comments 0 reactions 0 assignees View on GitHub
improvement P3 runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

When using a `HadoopInputFormatIO` the number of splits seems to be controlled by the underlying `InputFormat` which in turn determines the number of partitions and therefore parallelisation when running on Spark. It is possible to `Reshuffle` the data to compensate for data skew, but it _appears_ there is no way to grow the number of partitions. The `GroupCombineFunctions.reshuffle` seems to be the only place calling the Spark `repartition` and it uses the number of partitions from the original RDD.

Scenarios that would benefit from this:
- Increasing parallelisation for computationally heavy stages
- ETLs where the input partitions are dictated by the source while you wish to optimise the partitions for fast loading to the target sink
- Zip files (my case) where they are read in single threaded manner with a custom HadoopInputFormat and therefore get a single task for all stages

(It would be nice if a user could supply a partitioner too, to help dictate data locality)

Imported from Jira [BEAM-3022](https://issues.apache.org/jira/browse/BEAM-3022). Original Jira may contain additional context.
Reported by: timrobertson100.

Contributor guide

Open the contributing guide

Research direction

Start by reading the HadoopInputFormatIO path and GroupCombineFunctions.reshuffle, where the issue says Spark repartition currently uses the original RDD partition count. Define and implement the requested way for users to grow partition counts, then verify that computational stages and sink-loading stages can use the increased parallelism.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.