WriteFiles withRunnerDeterminedSharding for unbounded data doesn't work with merging windows
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently the implementation of `withRunnerDeterminedShardingUnbounded` uses a stateful DoFn to achieve the grouping and batching of the input elements, which doesn't support session windows. One possible way is to add another GBK prior to the stateful DoFn to first get session windows merged and reify the window before invoking the sateful DoFn.
Imported from Jira [BEAM-12040](https://issues.apache.org/jira/browse/BEAM-12040). Original Jira may contain additional context.
Reported by: sychen.
Contributor guide
Research direction
Start at the withRunnerDeterminedShardingUnbounded implementation and trace its stateful DoFn grouping and batching behavior for session windows. Reproduce the failure with merging windows, then evaluate the proposed preceding GBK and window reification approach. Done means WriteFiles with runner-determined sharding supports unbounded data with merging session windows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100