citusdata / citusdata/citus

(outer) recurring / (inner) non-recurring outer join support improvements

Open
#6,546 0 comments 0 reactions 1 assignee Claimed by @eaydingol View on GitHub
performance research
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Before #6512, joins between recurring rels and non-recurring (distributed) rels were not supported when the recurring rel is in the outer side and non-recurring one is in the inner side of join; and now we support such joins by recursively planning distributed side of the outer join, i.e., by converting the distributed side into an intermediate result / a group of intermediate results. Here, a recurring rel might be a reference table or an intermediate result created as a result of a complex subquery/cte/join; and a non-recurring rel means a distributed table or a subquery/cte/join that can be pushed down to the worker nodes.

For example, we support following join types as of #6512:

* \ LEFT JOIN \
* \ RIGHT JOIN \
* \ LEFT JOIN \
* \ FULL JOIN \

1- When the distributed side a join tree (as in the third example), we plan all the distributed tables _that make given join tree distributed_ individually. Meaning the third example becomes:
* \ LEFT JOIN
\

Ideally, we should handle such a sub join tree by moving it into a subquery "as a whole" but this implies that we need to rebuild the rtable and re-point all the Vars to the new rtable indexes, so we've not implemented that yet. Instead, we recursively plan all the distributed tables in that sub join tree. This is much more inefficient than the other approach (since we lose the opportunity to push-down the whole sub join tree into the workers) but was easier to implement.

See RecursivelyPlanDistributedJoinNode().

2- While the first item is specific to the joins where the distributed side is another join tree, and mostly involves implementation work; this second item is more like a generic improvement that can be made regardless of whatever the distributed side is (a distributed table, subquery or a join tree), and requires a prior research work. That is why this issue has a "research" label

Instead of simply recursively planning the "distributed rel", we could maybe come up with something smarter. This could either mean re-partitioning the recurring rel and then pushing the join down to the worker nodes; or doing some kind of pre/post processing before/after recursively planning distributed side of the join.

Given that we expect reference tables to be fact tables and smaller when compared to distributed tables, and joins between distributed tables are much more efficient than the ones between reference tables / intermediate results (due to parallelism), this could help improving the performance a lot but requires some more thinking.

See RecursivelyPlanRecurringTupleOuterJoinWalker().

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.