apache / apache/datafusion

Join dynamic filter pushdown through `FilterExec` with a projection can discard matching rows with duplicate column names

Closed
#25,262 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through a `FilterExec` that carries an embedded projection over an input with duplicate column names (here `a.id` and `b.id` from a nested join).

With the same data and SQL, disabling `datafusion.optimizer.enable_join_dynamic_filter_pushdown` returns one row; enabling it returns zero rows, without an error.

### To Reproduce

Run the following SQL in a fresh `datafusion-cli` session. The two `/tmp/` Parquet paths must not already exist.

```sql
SET datafusion.optimizer.join_reordering = false;

COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_filterprojection_a.parquet'
STORED AS PARQUET;

COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_filterprojection_b.parquet'
STORED AS PARQUET;

CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_filterprojection_a.parquet';

CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_filterprojection_b.parquet';

-- Returns one row: (x1, a1, x1).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;

SELECT s.id AS sid, a.id AS aid, b.id AS bid
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
WHERE (a.ty || '!') IS DISTINCT FROM b.id
)
ON s.id = b.id;

-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;

SELECT s.id AS sid, a.id AS aid, b.id AS bid
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
WHERE (a.ty || '!') IS DISTINCT FROM b.id
)
ON s.id = b.id;
```

### Expected behavior

_No response_

### Additional context

`FilterExec` resolves pushed-down filter columns by name instead of through its embedded projection, so the filter on `b.id` is applied to `a.id`. Found while working on #25244; the fix is included in #25259.

Contributor guide

Open the contributing guide

Research direction

Start by running the SQL reproduction in a fresh datafusion-cli session with join dynamic filter pushdown both disabled and enabled. Inspect FilterExec's handling of pushed-down filter columns when its embedded projection contains duplicate names, and compare the work against #25259; done means both settings retain the matching row without applying the b.id filter to a.id.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.