apache / apache/beam

DataflowPipelineJob should have an API that prints messages but doesn't wait for completion

Open
#17,974 0 comments 0 reactions 0 assignees View on GitHub
core improvement java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

DataflowPipelineJob has a method waitToFinish() that takes a handler for printing the job's output messages, AND waits for the job to finish, printing messages along the way using that handler.

However, there are cases when a caller would like to poll for the job's messages and print them, but would like to keep the job under the caller's control, rather than having to wait for it to complete.

E.g., one can imagine wanting to do the following "wait until a certain Aggregator in the job reaches a certain value, and then cancel the job, printing messages along the way". This is not possible with the current API, without copying code of waitToFinish().

Imported from Jira [BEAM-89](https://issues.apache.org/jira/browse/BEAM-89). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start by locating DataflowPipelineJob and reading waitToFinish(), especially how it polls for job messages and invokes its handler. The change is complete when callers can poll and print messages without waiting for job completion, while existing completion-waiting behavior remains covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.