apache / apache/beam

Backoff in the DirectRunner Monitor if no work is Available

Open
#18,070 0 comments 0 reactions 0 assignees View on GitHub
bug direct P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When a Pipeline has no elements available to process, the Monitor Runnable will be repeatedly scheduled. Given that there is no work to be done, this will loop over the steps in the transform looking for timers, and prompt the sources to perform additional work, even though there is no work to be done. This consumes the entirety of a single core.

Add a bounded backoff to rescheduling the monitor runnable if no work has been done since it last ran. This will reduce resource consumption on low-throughput Pipelines.

Imported from Jira [BEAM-690](https://issues.apache.org/jira/browse/BEAM-690). Original Jira may contain additional context.
Reported by: tgroh.

Contributor guide

Open the contributing guide

Research direction

Locate the DirectRunner Monitor Runnable and trace how it is rescheduled when no work is performed. Confirm the existing monitor and source-processing tests, then add bounded backoff for consecutive no-work runs and verify that low-throughput pipelines no longer reschedule continuously.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.