apache / apache/beam

Incomplete JobService API Semantics

Open
#19,284 0 comments 0 reactions 0 assignees View on GitHub
beam-model P3 tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The JobService API (beam_job_api.proto) allows for the possibility of never seeing messages or states with Get(State|Message)Stream. This is because the  Get(State|Message)Stream calls need to have the job id which can only be obtained from the RunJobResponse. But in order to see all messages/states the streams need to be opened before the job starts.

This is fine in Dataflow as the preparation_id \== job_id, but this is not true in Flink.

Fix is to modify the API to only keep a single id to be used between the preparation/run APIs. 

Consumers of the API will have to be modified to meet the new semantics.

Dev list thread (https://lists.apache.org/thread.html/3ace7585278c0545185fa4bb8d6975283d5c48c097e1bb2c2e18b9a2@%3Cdev.beam.apache.org%3E)

 

Imported from Jira [BEAM-6442](https://issues.apache.org/jira/browse/BEAM-6442). Original Jira may contain additional context.
Reported by: rohdesam.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.