Use Accumulable MetricsContainers in the DirectRunner
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Once PR 2649 is submitted, we should be able to use the accumulable functionality to clean up the DirectRunner. Since this runner updates metrics from multiple processing threads simultaneously, we will need to figure out how to either (1) make the accumulable behavior thread safe or (2) have a non-blocking way of performing the accumulation sequentially.
If we go with (2), we can put updated containers in a queue and then have a single thread trying to apply those updates.
https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java#L912)
Imported from Jira [BEAM-2186](https://issues.apache.org/jira/browse/BEAM-2186). Original Jira may contain additional context.
Reported by: bchambers.
Contributor guide
Research direction
Start with runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java around line 912, then review the accumulable functionality expected from PR 2649 and the original BEAM-2186 context. Determine whether thread-safe accumulation or a non-blocking sequential queue is appropriate; done means the DirectRunner uses accumulable metrics containers safely across processing threads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100