apache / apache/datafusion

Inconsistent `IS NOT NULL` filters for joins

Open
#6,432 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

In the Dask-SQL project, we have relied on DataFusion to create `IS NOT NULL` filters at the `TableScan` level whenever a column is involved in a join. However, it looks like recent changes may have removed this feature?

### To Reproduce

The query
```
SELECT d_col
FROM c_table
JOIN d_table ON d_col=c_col
```
has the `LogicalPlan`
```
Projection: d_table.d_col
Inner Join: Filter: d_table.d_col = c_table.c_col
TableScan: c_table projection=[c_col]
TableScan: d_table projection=[d_col]
```

### Expected behavior

It still works when we write the query with a `WHERE` clause.
```
SELECT d_col
FROM c_table, d_table WHERE d_col=c_col
```
produces
```
Projection: d_table.d_col
Inner Join: c_table.c_col= d_table.d_col
TableScan: c_table projection=[c_col], full_filters=[c_table.c_col IS NOT NULL]
TableScan: d_table projection=[d_col], full_filters=[d_table.d_col IS NOT NULL]
```

### Additional context

I'm not quite sure when this change was introduced and if so, why? Is this something that DataFusion would be willing to fix, or would it be preferred that Dask-SQL re-adds the optimizer rule on our side?

cc @ayushdg @jdye64

Contributor guide

Open the contributing guide

Research direction

Start by running the two SQL examples and comparing their LogicalPlan output, especially the TableScan full_filters. Trace the optimizer rule or DataFusion change that handles join predicates versus WHERE predicates. Done means the intended IS NOT NULL filters are consistently produced, or the ownership of that behavior between DataFusion and Dask-SQL is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.