apache / apache/beam

JobService#prepareJob should be one-to-many with JobService#run

Open
#18,650 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

More specifically, prepare should not require a job; instead, it should return endpoints for any pre-job work that the client must perform (e.g. staging artifacts). Once the client has completed any pre-job work, those endpoints should return tokens by which the results of the pre-job work can be referenced. #run should then consume those tokens and use them to ensure that those results are available during execution.

 

The simple example is that #prepare should return an artifact staging endpoint, and #run should take tokens returned by that endpoint that it can understand when executing the job. This permits multiple pipelines to be run with the same artifacts, among other benefits, and reduces the statefulness of the prepare-run call sequence.

Imported from Jira [BEAM-3536](https://issues.apache.org/jira/browse/BEAM-3536). Original Jira may contain additional context.
Reported by: tgroh.

Contributor guide

Open the contributing guide

Research direction

Start by reading the JobService#prepareJob and JobService#run entry points and the surrounding client interaction described in the issue. Trace how preparation currently depends on a job. Done means prepare can expose pre-job work and return reference tokens, while run consumes those tokens so multiple pipelines can use the same staged artifacts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.