apache / apache/beam

Range join in Beam

Open
#18,095 0 comments 1 reaction 0 assignees View on GitHub
new feature P3 sdk-ideas
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

1.We can support some data-driven trigger, so we need expose data in OnElementContext of onElement method.
2.We can support more flexible join, so we need expose buffer tag in TriggerContext, now this buffer tag is in SystemReduceFn.

for example: SELECT STREAM * FROM Orders AS o JOIN Shipments AS s
ON o.orderId = s.orderId AND s.rowtime BETWEEN o.rowtime AND o.rowtime **** INTERVAL '1' HOUR;

link: https://issues.apache.org/jira/browse/BEAM-101

Imported from Jira [BEAM-912](https://issues.apache.org/jira/browse/BEAM-912). Original Jira may contain additional context.
Reported by: lzljs3620320.

Contributor guide

Open the contributing guide

Research direction

Start with the linked Jira issues BEAM-101 and BEAM-912 for the missing context and range-join requirements. Then inspect OnElementContext, TriggerContext, and SystemReduceFn, which the issue identifies as the relevant entry points. Done means the required data and buffer tag are exposed so the described range join and data-driven trigger use cases can be supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.