Backoff in the DirectRunner Monitor if no work is Available
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When a Pipeline has no elements available to process, the Monitor Runnable will be repeatedly scheduled. Given that there is no work to be done, this will loop over the steps in the transform looking for timers, and prompt the sources to perform additional work, even though there is no work to be done. This consumes the entirety of a single core.
Add a bounded backoff to rescheduling the monitor runnable if no work has been done since it last ran. This will reduce resource consumption on low-throughput Pipelines.
Imported from Jira [BEAM-690](https://issues.apache.org/jira/browse/BEAM-690). Original Jira may contain additional context.
Reported by: tgroh.
Contributor guide
Research direction
Locate the DirectRunner Monitor Runnable and trace how it is rescheduled when no work is performed. Confirm the existing monitor and source-processing tests, then add bounded backoff for consecutive no-work runs and verify that low-throughput pipelines no longer reschedule continuously.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100