apache / apache/beam

OutputTimeFn and Accumulating Mode is Confusing

Open
#18,145 0 comments 0 reactions 0 assignees View on GitHub
beam-model bug P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

See [here]( https://github.com/tgroh/beam/commit/2238df334a368ce1a41e14ee616be954c5430c73) for an example pipeline

The Timestamp used by a pane does not change based on the accumulation mode of the windowing strategy - as a result, elements which have associated timestamps can not be safely reassigned to those timestamps after a GroupByKey if more than one pane could have been produced, regardless of the `OutputTimeFn`. The first example pipeline demonstrates two PCollections where the elements within the last PCollection cannot be reassigned to their timestamps, even though we are using `OutputTimeFn#outputAtEarliestInputTimestamp` and

When using a more complex windowing strategy like sessions, this is even more confusing - a session that spans more than one of the downstream windows but that is produced in multiple panes will over time be assigned to later and later windows as more panes are produced - thus, a pipeline that produces session windows and wishes to group the sessions by the point at which they started must only ever produce a single pane per session.

Imported from Jira [BEAM-1372](https://issues.apache.org/jira/browse/BEAM-1372). Original Jira may contain additional context.
Reported by: tgroh.

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files or tests; start by reviewing and running the example pipeline in commit 2238df334a368ce1a41e14ee616be954c5430c73. Trace how OutputTimeFn, windowing, and accumulating panes determine element timestamps, then identify the expected behavior for repeated panes and session windows. Done requires an agreed fix or documented behavior with coverage for the reported examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.