apache / apache/beam

Incremental join

Open
#18,001 0 comments 0 reactions 0 assignees View on GitHub
beam-model bug P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Consider a co-group by key over the two (streaming) collections:
l : PCollection\>
r : PCollection\>
Each processElement sees a K, Iterable and Iterable.

If the underlying trigger only allows a single PaneInfo.Timing.ON_TIME pane then it is trivial to calculate the traditional cross-product, including any of the inner/outer join combinations should Iterable or Iterable be empty.

However if the underlying trigger supports speculative (ie PaneInfo.Timing.EARLY) or late (ie PaneInfo.Timing.LATE) panes then the corresponding speculative output panes are awkward to compute.

(left_already_seen ******** new_left) X (right_already_seen ******** new_right)
\==
(left_already_seen X right_already_seen) ********
(new_left X right_already_seen) ********
(left_already_seen X new_right) ********
(new_left X new_right)

Currently the barrier between 'already seen' and 'new' must be maintained for left and right in per-window state. That suppresses some optimizations.

This bug is for finding a cleaner way to express this combinator.

Imported from Jira [BEAM-197](https://issues.apache.org/jira/browse/BEAM-197). Original Jira may contain additional context.
Reported by: mshields822.

Contributor guide

Open the contributing guide

Research direction

Start with the co-group operation over the two streaming PCollections and examine how speculative and late panes are represented by PaneInfo.Timing. Trace the per-window state that separates already-seen from new values. Done means a cleaner combinator expression that computes the required pane outputs without the current barrier.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.