Be able to specify the Spark Partitioner via the pipeline options
Open
new feature
P3
runners
spark
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
As we did for the StorageLevel, it would be great for an user to be able to provide the Spark partitionner via PipelineOptions.
Imported from Jira [BEAM-3192](https://issues.apache.org/jira/browse/BEAM-3192). Original Jira may contain additional context.
Reported by: jbonofre.
Contributor guide
Research direction
Start by locating the existing StorageLevel support in PipelineOptions and the Spark runner's handling of partitioners. Trace how the option would reach the Spark execution path, then check the original BEAM-3192 issue for missing context and define tests that verify a user-supplied partitioner is honored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100