Inconsistent `IS NOT NULL` filters for joins
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
In the Dask-SQL project, we have relied on DataFusion to create `IS NOT NULL` filters at the `TableScan` level whenever a column is involved in a join. However, it looks like recent changes may have removed this feature?
### To Reproduce
The query
```
SELECT d_col
FROM c_table
JOIN d_table ON d_col=c_col
```
has the `LogicalPlan`
```
Projection: d_table.d_col
Inner Join: Filter: d_table.d_col = c_table.c_col
TableScan: c_table projection=[c_col]
TableScan: d_table projection=[d_col]
```
### Expected behavior
It still works when we write the query with a `WHERE` clause.
```
SELECT d_col
FROM c_table, d_table WHERE d_col=c_col
```
produces
```
Projection: d_table.d_col
Inner Join: c_table.c_col= d_table.d_col
TableScan: c_table projection=[c_col], full_filters=[c_table.c_col IS NOT NULL]
TableScan: d_table projection=[d_col], full_filters=[d_table.d_col IS NOT NULL]
```
### Additional context
I'm not quite sure when this change was introduced and if so, why? Is this something that DataFusion would be willing to fix, or would it be preferred that Dask-SQL re-adds the optimizer rule on our side?
cc @ayushdg @jdye64
Contributor guide
Research direction
Start by running the two SQL examples and comparing their LogicalPlan output, especially the TableScan full_filters. Trace the optimizer rule or DataFusion change that handles join predicates versus WHERE predicates. Done means the intended IS NOT NULL filters are consistently produced, or the ownership of that behavior between DataFusion and Dask-SQL is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100