apache / apache/datafusion

Join dynamic filter pushdown through `ProjectionExec` can map duplicate output aliases to the wrong expression

Closed
#25,263 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through a `ProjectionExec` whose output has two columns with the same alias (here `a.id` and `b.id`, both named `id`).

With the same data and SQL, disabling `datafusion.optimizer.enable_join_dynamic_filter_pushdown` returns one row; enabling it returns zero rows, without an error.

### To Reproduce

Run the following SQL in a fresh `datafusion-cli` session. The two `/tmp/` Parquet paths must not already exist.

```sql
SET datafusion.optimizer.join_reordering = false;

COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_projection_a.parquet'
STORED AS PARQUET;

COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_projection_b.parquet'
STORED AS PARQUET;

CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_projection_a.parquet';

CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_projection_b.parquet';

-- Returns one row: (a1, a1, x1, x1!).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;

SELECT s.id, a.id, b.id, tag
FROM ta s
JOIN (
SELECT a.id, b.id, a.ty || '!' AS tag
FROM ta a JOIN tb b ON a.ty = b.id
)
ON s.id = a.id;

-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;

SELECT s.id, a.id, b.id, tag
FROM ta s
JOIN (
SELECT a.id, b.id, a.ty || '!' AS tag
FROM ta a JOIN tb b ON a.ty = b.id
)
ON s.id = a.id;
```

### Expected behavior

_No response_

### Additional context

`ProjectionExec` resolves pushed-down filter columns by output alias instead of by position, so with two outputs named `id` the filter on `b.id` is rewritten to `a.id`. Found while working on #25244; the fix is included in #25259.

Contributor guide

Open the contributing guide

Research direction

Reproduce the query in a fresh datafusion-cli session with join dynamic filter pushdown enabled and disabled, then inspect the ProjectionExec path that resolves pushed-down filter columns. Compare the behavior and the fix referenced in #25259; done means duplicate id aliases no longer discard the matching row and the regression is covered by a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.