apache / apache/beam

Refine specification for WindowFn.isCompatible()

Open
#18,104 0 comments 0 reactions 0 assignees View on GitHub
beam-model new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

`WindowFn#isCompatible` doesn't really have a spec. In practice, it is used primarily when flattening together multiple PCollections. All of the WindowFns must be compatible, and then just a single WindowFn is selected arbitrarily for the output PCollection.

In consequence, downstream of the Flatten, the merging behavior will be taken from this WindowFn.

Currently, there are some mismatches:

- Sessions with different gap durations _are_ compatible today, but probably shouldn't be since merging makes little sense. (The use of tiny proto-windows is an implementation detail anyhow)
- SlidingWindows and FixedWindows _could_ reasonably be compatible if they had the same duration, though it might be odd.

Either way, we should just nail down what we actually mean so we can arrive at a verdict in these cases.

Imported from Jira [BEAM-653](https://issues.apache.org/jira/browse/BEAM-653). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start by reading WindowFn#isCompatible and the flattening behavior described in the issue, focusing on how a single WindowFn is selected for the output PCollection. Compare the compatibility implications for Sessions, SlidingWindows, and FixedWindows, then document a settled specification and verify that the resulting behavior matches it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.