apache / apache/beam

WriteFiles withRunnerDeterminedSharding for unbounded data doesn't work with merging windows

Open
#20,928 0 comments 0 reactions 0 assignees View on GitHub
files improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently the implementation of `withRunnerDeterminedShardingUnbounded` uses a stateful DoFn to achieve the grouping and batching of the input elements, which doesn't support session windows. One possible way is to add another GBK prior to the stateful DoFn to first get session windows merged and reify the window before invoking the sateful DoFn. 

Imported from Jira [BEAM-12040](https://issues.apache.org/jira/browse/BEAM-12040). Original Jira may contain additional context.
Reported by: sychen.

Contributor guide

Open the contributing guide

Research direction

Start at the withRunnerDeterminedShardingUnbounded implementation and trace its stateful DoFn grouping and batching behavior for session windows. Reproduce the failure with merging windows, then evaluate the proposed preceding GBK and window reification approach. Done means WriteFiles with runner-determined sharding supports unbounded data with merging session windows.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.