apache / apache/beam

Spark runner should provide a default BoM

Open
#18,022 0 comments 0 reactions 0 assignees View on GitHub
improvement P3 runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

When an user defines a pipeline, he has to define the following dependencies in order to work with the Spark runner (in addition of the Spark runner artifact itself):

```


org.apache.spark
spark-core_2.10

${spark.version}



org.apache.spark
spark-streaming_2.10

${spark.version}



com.fasterxml.jackson.core

jackson-core
${jackson.version}



com.fasterxml.jackson.core

jackson-annotations
${jackson.version}



com.fasterxml.jackson.core

jackson-databind
${jackson.version}



com.fasterxml.jackson.module

jackson-module-scala_2.10
${jackson.version}

```

These dependencies should come as transitive dependencies from the Spark runner (as Flink or dataflow runners do). It means we should change the scope of the dependencies in the Spark runner pom to create a BoM (Bill of Material).

I will experiment a bit (with [~amitsela]).

Imported from Jira [BEAM-349](https://issues.apache.org/jira/browse/BEAM-349). Original Jira may contain additional context.
Reported by: jbonofre.

Contributor guide

Open the contributing guide

Research direction

Start in the Spark runner's Maven pom and compare its dependency declarations with the Flink and Dataflow runners. Verify how the listed Spark and Jackson dependencies are exposed to a pipeline and confirm that a pipeline using the Spark runner no longer needs to declare them separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.