Support executing MapReduce jobs in parallel.
Open
mapreduce
new feature
P3
runners
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently, the runner executes MR jobs sequentially with the topological order. This is very inefficient, it will be good to allow jobs to be executed in parallel if they don't depend on either other.
Imported from Jira [BEAM-2835](https://issues.apache.org/jira/browse/BEAM-2835). Original Jira may contain additional context.
Reported by: peihe0@gmail.com.
Contributor guide
Research direction
Start by reviewing the runner behavior described in the issue and the original Jira item BEAM-2835, which may contain additional context. Identify how jobs are currently followed in topological order and define completion as allowing independent MapReduce jobs to execute concurrently without violating dependencies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100