apache / apache/datafusion

Common subexpression elimination masks hides aggregations in physical explain output

Open
#19,684 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

When the common subexpression elimination deduplicates aggregations it can generate aliases for the common expression of the form `__common_expr_`. In the logical plan explain output this gets output as ` as __common_expr_`. In the physical plan explain output though only `__common_expr_` is printed. The actual expression corresponding to this alias is no longer visible. This makes the explain output hard to interpret.

### To Reproduce

Here's an example logic plan constructed using the data frame API. The problematic line is
```
AggregateExec: mode=Partial, gby=[idx@1 as idx], aggr=[__common_expr_1]
```

```
Logical plan
============
Projection: idx, agg, ord
Aggregate: groupBy=[[idx]], aggr=[[sum(column1) AS agg, sum(column1) AS ord]]
Projection: column1, column2, CASE WHEN column2 <= Int64(0) THEN Int64(0) WHEN column2 <= Int64(200) THEN Int64(1) WHEN column2 <= Int64(314) THEN Int64(3) ELSE Int64(4) END AS idx
Values: (Int64(1), Int64(100)), (Int64(2), Int64(200)), (Int64(3), Int64(314))

Optimized logical plan
======================
Projection: idx, __common_expr_1 AS agg, __common_expr_1 AS ord
Aggregate: groupBy=[[idx]], aggr=[[sum(column1) AS __common_expr_1]]
Projection: column1, CASE WHEN column2 <= Int64(0) THEN Int64(0) WHEN column2 <= Int64(200) THEN Int64(1) WHEN column2 <= Int64(314) THEN Int64(3) ELSE Int64(4) END AS idx
Values: (Int64(1), Int64(100)), (Int64(2), Int64(200)), (Int64(3), Int64(314))

Physical plan
=============
ProjectionExec: expr=[idx@0 as idx, __common_expr_1@1 as agg, __common_expr_1@1 as ord]
AggregateExec: mode=FinalPartitioned, gby=[idx@0 as idx], aggr=[__common_expr_1]
RepartitionExec: partitioning=Hash([idx@0], 10), input_partitions=1
AggregateExec: mode=Partial, gby=[idx@1 as idx], aggr=[__common_expr_1]
ProjectionExec: expr=[column1@0 as column1, CASE WHEN column2@1 <= 0 THEN 0 WHEN column2@1 <= 200 THEN 1 WHEN column2@1 <= 314 THEN 3 ELSE 4 END as idx]
DataSourceExec: partitions=1, partition_sizes=[1]
```

### Expected behavior

Rather than
```
AggregateExec: mode=Partial, gby=[idx@1 as idx], aggr=[__common_expr_1]
```
the explain output should show
```
AggregateExec: mode=Partial, gby=[idx@1 as idx], aggr=[sum(column1@0) as __common_expr_1]
```
similarly to how the group by expression are printed.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the physical-plan explain formatting for AggregateExec and the __common_expr_ aliases, then reproduce the example in the issue. The change is done when the partial and final aggregate output includes the underlying aggregation expression, such as sum(column1@0) as __common_expr_1, while preserving the existing group-by formatting.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.