apache / apache/beam

Extend the SparkContextOptions

Open
#18,318 0 comments 0 reactions 0 assignees View on GitHub
improvement P3 runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently, it's possible to use the `SparkContextOptions` to "pass" a custom `SparkContext` to the Spark runner.

However, more than constructing a complete `SparkContext` outside of the runner, an user may want to "tweak" the context created internally by the runner.
We could "extend" the `SparkContextOptions` to pass additional options used to construct and populate the `JavaSparkContext` like security, dependencies management, additional properties.

Imported from Jira [BEAM-2381](https://issues.apache.org/jira/browse/BEAM-2381). Original Jira may contain additional context.
Reported by: jbonofre.

Contributor guide

Open the contributing guide

Research direction

Start by locating SparkContextOptions and the Spark runner, then trace how the runner constructs and populates JavaSparkContext. Determine how security, dependency-management, and additional properties should be represented and passed through; done means callers can tweak the internally created context without constructing a complete SparkContext themselves.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.