apache / apache/beam

Add 'garbage collection' hold when receive late element

Open
#17,984 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

We currently add a 'garbage collection' hold in WatermarkHold (invoked via ReduceFnRunner) if the closing behavior is FIRE_ALWAYS. This means an element which has come in too late for a data holds and an end-of-window hold may end up setting no hold at all. As a result, the eventual pane containing that element may end up dropped as being too late.

Imported from Jira [BEAM-311](https://issues.apache.org/jira/browse/BEAM-311). Original Jira may contain additional context.
Reported by: mshields822.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the interaction between WatermarkHold and ReduceFnRunner when closing behavior is FIRE_ALWAYS, focusing on late elements and existing hold selection. Done means a late element retains a garbage-collection hold alongside the relevant end-of-window behavior so its eventual pane is not dropped as too late; add or update the corresponding tests found during that trace.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.