apache / apache/beam

The DirectRunner can complete before reporting the results of all ModelEnforcements

Open
#18,097 0 comments 0 reactions 0 assignees View on GitHub
bug direct P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The termination condition in the ExecutorServiceParallelExecutor.MonitorRunnable checks to see if the evaluation context is done, among other things. Due to the signatures of ModelEnforcement, output is committed before ModelEnforcement runs to ensure that the enforcement has access to the final elements. If the transform that performs ModelEnforcement completes, and all PTransforms are done, the pipeline can shutdown before model enforcement fails.

Imported from Jira [BEAM-819](https://issues.apache.org/jira/browse/BEAM-819). Original Jira may contain additional context.
Reported by: tgroh.

Contributor guide

Open the contributing guide

Research direction

Start with DirectRunner's ExecutorServiceParallelExecutor.MonitorRunnable and trace how its termination condition observes the evaluation context and completed PTransforms. Review the ModelEnforcement execution path, then verify that the DirectRunner remains alive until all ModelEnforcements have reported their results, including failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.