Incremental join
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Consider a co-group by key over the two (streaming) collections:
l : PCollection\>
r : PCollection\>
Each processElement sees a K, Iterable and Iterable.
If the underlying trigger only allows a single PaneInfo.Timing.ON_TIME pane then it is trivial to calculate the traditional cross-product, including any of the inner/outer join combinations should Iterable or Iterable be empty.
However if the underlying trigger supports speculative (ie PaneInfo.Timing.EARLY) or late (ie PaneInfo.Timing.LATE) panes then the corresponding speculative output panes are awkward to compute.
(left_already_seen ******** new_left) X (right_already_seen ******** new_right)
\==
(left_already_seen X right_already_seen) ********
(new_left X right_already_seen) ********
(left_already_seen X new_right) ********
(new_left X new_right)
Currently the barrier between 'already seen' and 'new' must be maintained for left and right in per-window state. That suppresses some optimizations.
This bug is for finding a cleaner way to express this combinator.
Imported from Jira [BEAM-197](https://issues.apache.org/jira/browse/BEAM-197). Original Jira may contain additional context.
Reported by: mshields822.
Contributor guide
Research direction
Start with the co-group operation over the two streaming PCollections and examine how speculative and late panes are represented by PaneInfo.Timing. Trace the per-window state that separates already-seen from new values. Done means a cleaner combinator expression that computes the required pane outputs without the current barrier.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100