apache / apache/datafusion

Join DynamicFilter enhancements

Open
#16,973 6 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Tracking issue for improvements to dynamic filters w/ joins

- [ ] Generate filters for more join types
- [ ] Allow parent filters to pass through in more cases (https://github.com/apache/datafusion/pull/20192)
- [ ] Support nested joins (https://github.com/apache/datafusion/issues/20214)

#### Generate filters for more join types

In https://github.com/apache/datafusion/pull/16954 we are only going to support inner joins because of complexities with other join types.

This would benefit cases such as:

```sql
COPY (SELECT 1 AS c1) TO 't1.parquet';
COPY (SELECT 1 AS c1, 2 AS c2) TO 't2.parquet';

CREATE EXTERNAL TABLE t1 (
c1 INT NOT NULL
)
STORED AS PARQUET
LOCATION 't1.parquet';

CREATE EXTERNAL TABLE t2 (
c1 INT NOT NULL,
c2 INT NOT NULL
)
STORED AS PARQUET
LOCATION 't2.parquet';

select *
from t1
left join t2 using (c1)
order by t2.c2
limit 100;
```

### Describe the solution you'd like

An implementation similar to that of the logical optimizer that accounts for the edge cases and allows some filters to be pushed down.

Contributor guide

Open the contributing guide

Research direction

Start by reading PR 16954, PR 20192, and issue 20214, then use the SQL join example and the tracking checklist to understand the current dynamic-filter limits. Done means extending filter generation and propagation to additional join cases, including nested joins, with the relevant edge cases covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.