apache / apache/datafusion

Projection Expr is collapsed into a single Column string when GROUP BY is present, losing the original Expr

Open
#19,353 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

When a query contains a `GROUP BY` clause, DataFusion collapses complex projection expressions into a single Column whose name is a stringified version of the full expression. This causes the original expression tree (`Expr`) to be lost, existing only within the expression of the `AggregatePlan`

For the same projection expression:

Without `GROUP BY`: the projection remains a fully composed `Expr` (e.g. Cast, ScalarFunction, etc.).

With `GROUP BY`: the projection is replaced with `Alias(Column())`.

This prevents analyzer or optimizer rules from inspecting or rewriting the original expression.

### To Reproduce

```rust
use std::sync::Arc;

use datafusion::{
arrow::{
array::{Int32Array, RecordBatch, StringArray, TimestampSecondArray},
datatypes::{DataType, Field, Schema, TimeUnit},
},
common::tree_node::{Transformed, TreeNode},
datasource::MemTable,
execution::{SessionStateBuilder, context::SessionContext},
logical_expr::LogicalPlan,
};

fn create_test_session() -> SessionContext {
let state = SessionStateBuilder::new().with_default_features().build();
let ctx = SessionContext::new_with_state(state);

let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("name", DataType::Utf8, false),
Field::new(
"_modifiedDateTime",
DataType::Timestamp(TimeUnit::Second, None),
false,
),
]));

let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(Int32Array::from(vec![1, 2, 3])),
Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])),
Arc::new(TimestampSecondArray::from(vec![
chrono::Utc::now().timestamp(),
chrono::Utc::now().timestamp(),
chrono::Utc::now().timestamp(),
])),
],
)
.unwrap();

let table = MemTable::try_new(schema, vec![vec![batch]]).unwrap();
ctx.register_table("users", Arc::new(table)).unwrap();

ctx
}

#[tokio::main]
async fn main() {
let ctx = create_test_session();
let sql = r#"
SELECT
CAST(
DATE_FORMAT(
CAST(
TO_DATE(
DATE_FORMAT(
CAST(`_modifiedDateTime` AS TIMESTAMP),
'%Y-%m-%dT%H:%i:%S'
),
'%Y-%m-%dT%H:%i:%S'
) AS TIMESTAMP
),
'%Y-%m-%d %H:%i:%s'
) AS TIMESTAMP
) AS qt_5anmnq5myd
FROM users group by qt_5anmnq5myd;
"#;

let plan = ctx.state().create_logical_plan(sql).await.unwrap();

plan.transform(|p| match p {
LogicalPlan::Projection(ref projection) => {
println!("DEBUG {:?}", projection.expr);
Ok(Transformed::no(p))
}
_ => Ok(Transformed::no(p)),
})
.unwrap();
}

```

### Suggested Behavior

The projection expression should remain a structured `Expr` tree regardless of whether a `GROUP BY` clause is present.

Specifically:

The logical plan should preserve the original composed expression (Cast, ScalarFunction, etc.).

Analyzer and optimizer rules should be able to traverse and rewrite the projection expression consistently for both grouped and non-grouped queries.

### Additional context

By tracing in datafusion/sql/src/select.rs, the expression is rewritten during aggregation planning:

```
if !group_by_exprs.is_empty() || !aggr_exprs.is_empty() {
self.aggregate(...)
}
```

Before aggregation: `select_exprs` contains the fully composed `Expr`.

After aggregation: `select_exprs_post_aggr` contains `Alias(Column())`, and the schema contains only this single stringified column.

This causes problems for downstream `AnalyzerRules`. In our case, we have an analyzer that rewrites schemas to match a remote database schema, but for grouped queries we can no longer identify or rewrite the original projection expression.

We implemented a workaround that attempts to reconstruct the projection expression by matching it back to the `GROUP BY` expression in the `AggregatePlan` and re-stringifying it, but this is fragile and not ideal.

A possible improvement would be to delay collapsing projection expressions into stringified Columns until later(e.g. after analyzer rules or during optimization planning), so expression structure is preserved during logical analysis.

CC: @backkem

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/sql/src/select.rs and trace the aggregate path reached when GROUP BY is present, comparing select_exprs with select_exprs_post_aggr. Reproduce the query through SessionContext::create_logical_plan and inspect the logical plan. Done means grouped projections preserve their structured Expr trees so analyzer and optimizer rules can traverse them consistently with non-grouped projections.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.