apache / apache/beam

Support conjunction clause when it's only equi-join

Open
#19,176 0 comments 0 reactions 0 assignees View on GitHub
dsl P3 sql sub-task
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

conjunction_clause: function_call(function_parameter, ...) | field_access | column
function_parameter: function_call | field_access

In Beam, equi-join is implemented by CoGBK, which requires both join inputs (assume binary join) to build PCollection of KV, where the key is join key.

For equi-join, conjunction clause is essentially an equation. In order to build KV, it requires that columns from different sides of equation should come from different join input. For example, a **** b = 2 cannot be used to build join key but a = 2 - b can. So rewriting is required for clauses when it does not satisfy this property.

It also implies that not every clause is rewritable. Say the clause is f(a, b) = 3, in which a is from left input and b is from right input. If this function f is not splittable, such that we cannot move a or b to right side of equation, then we cannot support this clause in BeamSQL's join.

Imported from Jira [BEAM-7151](https://issues.apache.org/jira/browse/BEAM-7151). Original Jira may contain additional context.
Reported by: amaliujia.
Subtask of issue #19175

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.