Reordering of predicates and filters (pushing up and/or pushing down) in JOINs to get the most performant plan
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
Currently, Druid does not support pushing down predicates (condition and filter) pass a Join (i.e. into Join's children). Druid only supports pushing predicates into the join if they originated from above the join. Hence, the location of predicates and filters in your Druid SQL is very important.
Since Druid cannot reliably push up / push down filters smartly (pushing up is NOT always the best and pushing down is NOT always the best but Druid is not smart enough to figure this out), the PR https://github.com/apache/druid/pull/9773 disabled pushing down predicates (condition and filter) pass a Join (i.e. into Join's children) altogether.
We should rethink how we do the following:
1) Apply and write rules for JOIN (especially around pushing up/down predicates such as Rule that pushes predicates from a Filter into the Join below them and Rule that pushes predicates in a Join into the inputs to the join)
2) how we calculate cost for JOIN.
### Motivation
Druid should be able to smartly reorder predicates and filters (pushing up and/or pushing down) in JOINS to get the most performant plan.
Contributor guide
Research direction
Start by reviewing PR 9773 and the JOIN predicate pushdown rules named in the issue, then examine how JOIN cost is calculated. The work is done when Druid can smartly reorder predicates and filters around JOINs to produce more performant plans.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100