Extend the SparkContextOptions
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently, it's possible to use the `SparkContextOptions` to "pass" a custom `SparkContext` to the Spark runner.
However, more than constructing a complete `SparkContext` outside of the runner, an user may want to "tweak" the context created internally by the runner.
We could "extend" the `SparkContextOptions` to pass additional options used to construct and populate the `JavaSparkContext` like security, dependencies management, additional properties.
Imported from Jira [BEAM-2381](https://issues.apache.org/jira/browse/BEAM-2381). Original Jira may contain additional context.
Reported by: jbonofre.
Contributor guide
Research direction
Start by locating SparkContextOptions and the Spark runner, then trace how the runner constructs and populates JavaSparkContext. Determine how security, dependency-management, and additional properties should be represented and passed through; done means callers can tweak the internally created context without constructing a complete SparkContext themselves.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100