Add 'garbage collection' hold when receive late element
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
We currently add a 'garbage collection' hold in WatermarkHold (invoked via ReduceFnRunner) if the closing behavior is FIRE_ALWAYS. This means an element which has come in too late for a data holds and an end-of-window hold may end up setting no hold at all. As a result, the eventual pane containing that element may end up dropped as being too late.
Imported from Jira [BEAM-311](https://issues.apache.org/jira/browse/BEAM-311). Original Jira may contain additional context.
Reported by: mshields822.
Contributor guide
Research direction
Start by tracing the interaction between WatermarkHold and ReduceFnRunner when closing behavior is FIRE_ALWAYS, focusing on late elements and existing hold selection. Done means a late element retains a garbage-collection hold alongside the relevant end-of-window behavior so its eventual pane is not dropped as too late; add or update the corresponding tests found during that trace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100