apache / apache/beam

Improve interplay between PushbackSideInputRunner and GroupAlsoByWindowViaWindowSetDoFn

Open
#18,358 0 comments 0 reactions 0 assignees View on GitHub
core flink improvement P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

This originated from a discussion on a PR: https://github.com/apache/beam/pull/2235

`GroupAlsoByWindowViaWindowSetDoFn`/`GroupAlsoByWindowViaWindowSetNewDoFn` and `PushbackSideInputDoFnRunner` don't work well together and we manually need to explode windows in `FlinkStreamingTransformTranslators.ToKeyedWorkItem` because of this:

- `GroupAlsoByWindowViaWindowSetDoFn` is a `DoFn, KV>` so you have to push in `KeyedWorkItem`. These themselves contain `WindowedValue` (or timers).
- For executing a `DoFn` we use a `DoFnRunner`. For our problem the interesting case is using a `PushbackSideInputDoFnRunner`. The interesting method is `processElementInReadyWindows(WindowedValue elem)` where `InputT` is the input type of the `DoFn` which, for the windowing case, is `KeyedWorkItem` (from above). The actual expanded type signature is thus `processElementInReadyWindows(WindowedValue> elem)` where the keyed work items again contain `WindowedValues` (again, from above).
I think the `PushbackSideInputDoFnRunner` was not initially meant for executing `GroupAlsoByWindowViaWindowSetDoFns`.

Imported from Jira [BEAM-1850](https://issues.apache.org/jira/browse/BEAM-1850). Original Jira may contain additional context.
Reported by: aljoscha.

Contributor guide

Open the contributing guide

Research direction

Start with PushbackSideInputDoFnRunner.processElementInReadyWindows, GroupAlsoByWindowViaWindowSetDoFn and GroupAlsoByWindowViaWindowSetNewDoFn, then inspect FlinkStreamingTransformTranslators.ToKeyedWorkItem. Read the discussion in PR 2235 and the imported BEAM-1850 context to determine the intended interaction. Done should remove the need to manually explode windows for this combination.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.