apache / apache/beam

When and how can merging windows "shrink" or "grow"?

Open
#18,102 0 comments 0 reactions 0 assignees View on GitHub
beam-model new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The primary example of a merging window today is `Sessions` by gap duration, in which the merged window is the interval enclosure / span of the windows being merged.

However, another reasonable abstract use case is a session identified by id with an explicit end event. We might consider that the session ends with no gap duration after the end event. In this case, the merged window may be smaller than the enclosure of the sub-windows. Sometimes this has been referred to as "merging shrinks the window".

Perhaps the only requirement is that the merged window contains the timestamps of the data therein, but we should document this clearly. The current spec is ["Does whatever merging is necessary"](https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/WindowFn.java#L106)

There are repercussions for triggers, some documented in the [trigger design doc](https://s.apache.org/beam-triggers): With nonzero allowed lateness, `Sessions` by gap duration can switch a trigger from ON_TIME or LATE behavior back to speculative behavior, or cause another ON_TIME firing. Conversely, sessions with abrupt termination/shrinking may have that behavior _as well as_ an ON_TIME and subsequent LATE firings due only to the merging (this already works properly).

Imported from Jira [BEAM-654](https://issues.apache.org/jira/browse/BEAM-654). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start with the merging semantics in sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/WindowFn.java and review the linked trigger design document. Compare gap-based Sessions with explicitly terminated sessions and use Jira BEAM-654 for additional context. Done means the specification clearly explains when merged windows may shrink or grow and the resulting trigger behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.