Join dynamic filter pushdown through `FilterExec` with a projection can discard matching rows with duplicate column names
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through a `FilterExec` that carries an embedded projection over an input with duplicate column names (here `a.id` and `b.id` from a nested join).
With the same data and SQL, disabling `datafusion.optimizer.enable_join_dynamic_filter_pushdown` returns one row; enabling it returns zero rows, without an error.
### To Reproduce
Run the following SQL in a fresh `datafusion-cli` session. The two `/tmp/` Parquet paths must not already exist.
```sql
SET datafusion.optimizer.join_reordering = false;
COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_filterprojection_a.parquet'
STORED AS PARQUET;
COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_filterprojection_b.parquet'
STORED AS PARQUET;
CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_filterprojection_a.parquet';
CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_filterprojection_b.parquet';
-- Returns one row: (x1, a1, x1).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;
SELECT s.id AS sid, a.id AS aid, b.id AS bid
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
WHERE (a.ty || '!') IS DISTINCT FROM b.id
)
ON s.id = b.id;
-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
SELECT s.id AS sid, a.id AS aid, b.id AS bid
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
WHERE (a.ty || '!') IS DISTINCT FROM b.id
)
ON s.id = b.id;
```
### Expected behavior
_No response_
### Additional context
`FilterExec` resolves pushed-down filter columns by name instead of through its embedded projection, so the filter on `b.id` is applied to `a.id`. Found while working on #25244; the fix is included in #25259.
Contributor guide
Research direction
Start by running the SQL reproduction in a fresh datafusion-cli session with join dynamic filter pushdown both disabled and enabled. Inspect FilterExec's handling of pushed-down filter columns when its embedded projection contains duplicate names, and compare the work against #25259; done means both settings retain the matching row without applying the b.id filter to a.id.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100