apache / apache/datafusion

A fixed GROUP BY column is incorrectly used to decide when groups are complete

Closed
#25,018 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

When a `WHERE` condition limits a `GROUP BY` column to one value, we have the pieces to know that the column is fixed. However, aggs currently treats that column as if a change in its value could show that earlier groups are complete.

Since the column is fixed because of the filter, it cannot provide such a boundary.

eg:
```sql
SELECT key, market, COUNT(*)
FROM t
WHERE market = 'US'
GROUP BY key, market;
```

Today we currently report the agg as partially ordered using `market`. For large inputs, this blocks releasing finished groups early

### To Reproduce

```sql

EXPLAIN
WITH t(key, market) AS (
VALUES
(1, 'US'),
(2, 'US'),
(1, 'US')
)
SELECT key, market, COUNT(*)
FROM t
WHERE market = 'US'
GROUP BY key, market;
```

As of today, this produces

```text
AggregateExec: mode=Partial,
gby=[key@0 as key, market@1 as market],
ordering_mode=PartiallySorted([1])
```

Every row has `market = 'US'`, so this column never changes and cannot show that a group has finished.

### Expected behavior

Ideally should ignore fixed grouping columns when deciding whether earlier groups are complete. For the example above, agg should not use `market` as a group completion boundary. If another grouping column is genuinely ordered, we should still use that column

### Additional context

#24697 is somewhat related to this fix. It separates input ordering from the question of whether a group is complete. A fixed column can be considered ordered, but it should not be used as that a group is complete.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the EXPLAIN query and inspect AggregateExec's ordering and group-completion decisions, using the distinction described in issue #24697. Done means fixed GROUP BY columns such as market are excluded as completion boundaries while genuinely ordered grouping columns remain usable.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.