Join DynamicFilter enhancements
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Tracking issue for improvements to dynamic filters w/ joins
- [ ] Generate filters for more join types
- [ ] Allow parent filters to pass through in more cases (https://github.com/apache/datafusion/pull/20192)
- [ ] Support nested joins (https://github.com/apache/datafusion/issues/20214)
#### Generate filters for more join types
In https://github.com/apache/datafusion/pull/16954 we are only going to support inner joins because of complexities with other join types.
This would benefit cases such as:
```sql
COPY (SELECT 1 AS c1) TO 't1.parquet';
COPY (SELECT 1 AS c1, 2 AS c2) TO 't2.parquet';
CREATE EXTERNAL TABLE t1 (
c1 INT NOT NULL
)
STORED AS PARQUET
LOCATION 't1.parquet';
CREATE EXTERNAL TABLE t2 (
c1 INT NOT NULL,
c2 INT NOT NULL
)
STORED AS PARQUET
LOCATION 't2.parquet';
select *
from t1
left join t2 using (c1)
order by t2.c2
limit 100;
```
### Describe the solution you'd like
An implementation similar to that of the logical optimizer that accounts for the edge cases and allows some filters to be pushed down.
Contributor guide
Research direction
Start by reading PR 16954, PR 20192, and issue 20214, then use the SQL join example and the tracking checklist to understand the current dynamic-filter limits. Done means extending filter generation and propagation to additional join cases, including nested joins, with the relevant edge cases covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100