Incomplete JobService API Semantics
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The JobService API (beam_job_api.proto) allows for the possibility of never seeing messages or states with Get(State|Message)Stream. This is because the Get(State|Message)Stream calls need to have the job id which can only be obtained from the RunJobResponse. But in order to see all messages/states the streams need to be opened before the job starts.
This is fine in Dataflow as the preparation_id \== job_id, but this is not true in Flink.
Fix is to modify the API to only keep a single id to be used between the preparation/run APIs.
Consumers of the API will have to be modified to meet the new semantics.
Dev list thread (https://lists.apache.org/thread.html/3ace7585278c0545185fa4bb8d6975283d5c48c097e1bb2c2e18b9a2@%3Cdev.beam.apache.org%3E)
Imported from Jira [BEAM-6442](https://issues.apache.org/jira/browse/BEAM-6442). Original Jira may contain additional context.
Reported by: rohdesam.
Contributor guide
Assessment
This issue has not been assessed yet.