apache / apache/datafusion

Join dynamic filter pushdown through `AggregateExec` can confuse same-named grouping columns

Closed
#25,264 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through an `AggregateExec` that groups on two columns with the same name (here `a.id` and `b.id` from a nested join).

With the same data and SQL, disabling `datafusion.optimizer.enable_join_dynamic_filter_pushdown` returns one row; enabling it returns zero rows, without an error.

### To Reproduce

Run the following SQL in a fresh `datafusion-cli` session. The two `/tmp/` Parquet paths must not already exist.

```sql
SET datafusion.optimizer.join_reordering = false;

COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_aggregate_a.parquet'
STORED AS PARQUET;

COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_aggregate_b.parquet'
STORED AS PARQUET;

CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_aggregate_a.parquet';

CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_aggregate_b.parquet';

-- Returns one row: (x1, a1, x1).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;

SELECT s.id, a.id, b.id
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
GROUP BY a.id, b.id
)
ON s.id = b.id;

-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;

SELECT s.id, a.id, b.id
FROM tb s
JOIN (
SELECT a.id, b.id
FROM ta a LEFT JOIN tb b ON a.ty = b.id
GROUP BY a.id, b.id
)
ON s.id = b.id;
```

### Expected behavior

_No response_

### Additional context

`AggregateExec` resolves pushed-down filter columns by name instead of by grouping position, so the filter on the grouping column `b.id` is applied to `a.id`. Found while working on #25244; the fix is included in #25259.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the query in a fresh datafusion-cli session with join dynamic filter pushdown enabled and disabled, then inspect AggregateExec's handling of pushed-down filter columns. Done means the enabled query retains the matching row and the regression is covered, but the issue says the fix is already included in #25259.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.