DataflowPipelineJob should have an API that prints messages but doesn't wait for completion
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
DataflowPipelineJob has a method waitToFinish() that takes a handler for printing the job's output messages, AND waits for the job to finish, printing messages along the way using that handler.
However, there are cases when a caller would like to poll for the job's messages and print them, but would like to keep the job under the caller's control, rather than having to wait for it to complete.
E.g., one can imagine wanting to do the following "wait until a certain Aggregator in the job reaches a certain value, and then cancel the job, printing messages along the way". This is not possible with the current API, without copying code of waitToFinish().
Imported from Jira [BEAM-89](https://issues.apache.org/jira/browse/BEAM-89). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start by locating DataflowPipelineJob and reading waitToFinish(), especially how it polls for job messages and invokes its handler. The change is complete when callers can poll and print messages without waiting for job completion, while existing completion-waiting behavior remains covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100