citusdata / citusdata/citus

Compound join conditions and IsJoinClause

Open
#753 3 comments 0 reactions 0 assignees View on GitHub
technical debt
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

We recently ran into an issue related to compound join conditions (#58) in a private Slack channel. @robin900 looked deeper into the issue and noted that fixing the issue is tricky due to multiple code paths that do join planning. I'm copy/pasting his notes from the Slack chat below.
##

The two most common patterns for us, when there is a compound join condition, are:

1) `a JOIN b on a.dist_column = b.dist_column AND a.some_tstz <@ b.some_tstzrange`
2) `a JOIN b on a.dist_column = b.dist_column AND a.tstz < b.tstz`

We also have a rare `a JOIN b on a.dist_column = b.dist_column AND a.ctid < b.ctid` but we can work around the ctid problem.

My understanding, after reviewing the citus code, is that it would take significant refactoring to evaluate the list of clauses all at once, instead of `IsJoinClause()` called in series on a `clauseList`.

I note that the dummy join `a.fk_account_id = b.fk_account_id AND b.created_at = lower(a.during)` works merely because `IsJoinClause` returns true for both clauses.

They use `=` and left and right expr reference different tables (different `->varno`).

Reading the citus code, i see a confusion of what question the function `IsJoinClause` answers. This confusion leads to my confusion.

I see 2, perhaps 3, uses of the function:
1. In multi_join_order.c, to determine fixed join order for implicit joins; plus inside `ApplicableJoinClauses` where it’s just checking whether the left table and right table in each clause are applicable, and it’s not clear that equality operator is necessary.
2. In multi_logical_planner.c, to determine which clauses should participate in joins but not clear if equality operator is necessary for the logic to work.
3. A rough equivalent of `IsJoinClause` for lateral join optimization: https://github.com/citusdata/citus/blob/aa15043b0905a1fdba7f649d4986204c5efdf1e2/src/backend/distributed/planner/multi_logical_optimizer.c#L3431

Contributor guide

Open the contributing guide

Research direction

Start by tracing IsJoinClause and ApplicableJoinClauses in multi_join_order.c and multi_logical_planner.c, then compare the related lateral join logic in multi_logical_optimizer.c around the referenced line. Determine which callers require equality operators and how compound clause lists are handled. Done means the join-planning paths consistently support the reported compound conditions, with tests or other verification described by the implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.