Spark runner should provide a default BoM
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
When an user defines a pipeline, he has to define the following dependencies in order to work with the Spark runner (in addition of the Spark runner artifact itself):
```
org.apache.spark
spark-core_2.10
${spark.version}
org.apache.spark
spark-streaming_2.10
${spark.version}
com.fasterxml.jackson.core
jackson-core
${jackson.version}
com.fasterxml.jackson.core
jackson-annotations
${jackson.version}
com.fasterxml.jackson.core
jackson-databind
${jackson.version}
com.fasterxml.jackson.module
jackson-module-scala_2.10
${jackson.version}
```
These dependencies should come as transitive dependencies from the Spark runner (as Flink or dataflow runners do). It means we should change the scope of the dependencies in the Spark runner pom to create a BoM (Bill of Material).
I will experiment a bit (with [~amitsela]).
Imported from Jira [BEAM-349](https://issues.apache.org/jira/browse/BEAM-349). Original Jira may contain additional context.
Reported by: jbonofre.
Contributor guide
Research direction
Start in the Spark runner's Maven pom and compare its dependency declarations with the Flink and Dataflow runners. Verify how the listed Spark and Jackson dependencies are exposed to a pipeline and confirm that a pipeline using the Spark runner no longer needs to declare them separately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100