Refine specification for WindowFn.isCompatible()
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
`WindowFn#isCompatible` doesn't really have a spec. In practice, it is used primarily when flattening together multiple PCollections. All of the WindowFns must be compatible, and then just a single WindowFn is selected arbitrarily for the output PCollection.
In consequence, downstream of the Flatten, the merging behavior will be taken from this WindowFn.
Currently, there are some mismatches:
- Sessions with different gap durations _are_ compatible today, but probably shouldn't be since merging makes little sense. (The use of tiny proto-windows is an implementation detail anyhow)
- SlidingWindows and FixedWindows _could_ reasonably be compatible if they had the same duration, though it might be odd.
Either way, we should just nail down what we actually mean so we can arrive at a verdict in these cases.
Imported from Jira [BEAM-653](https://issues.apache.org/jira/browse/BEAM-653). Original Jira may contain additional context.
Reported by: kenn.
Contributor guide
Research direction
Start by reading WindowFn#isCompatible and the flattening behavior described in the issue, focusing on how a single WindowFn is selected for the output PCollection. Compare the compatibility implications for Sessions, SlidingWindows, and FixedWindows, then document a settled specification and verify that the resulting behavior matches it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100