Projection Expr is collapsed into a single Column string when GROUP BY is present, losing the original Expr
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
When a query contains a `GROUP BY` clause, DataFusion collapses complex projection expressions into a single Column whose name is a stringified version of the full expression. This causes the original expression tree (`Expr`) to be lost, existing only within the expression of the `AggregatePlan`
For the same projection expression:
Without `GROUP BY`: the projection remains a fully composed `Expr` (e.g. Cast, ScalarFunction, etc.).
With `GROUP BY`: the projection is replaced with `Alias(Column())`.
This prevents analyzer or optimizer rules from inspecting or rewriting the original expression.
### To Reproduce
```rust
use std::sync::Arc;
use datafusion::{
arrow::{
array::{Int32Array, RecordBatch, StringArray, TimestampSecondArray},
datatypes::{DataType, Field, Schema, TimeUnit},
},
common::tree_node::{Transformed, TreeNode},
datasource::MemTable,
execution::{SessionStateBuilder, context::SessionContext},
logical_expr::LogicalPlan,
};
fn create_test_session() -> SessionContext {
let state = SessionStateBuilder::new().with_default_features().build();
let ctx = SessionContext::new_with_state(state);
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int32, false),
Field::new("name", DataType::Utf8, false),
Field::new(
"_modifiedDateTime",
DataType::Timestamp(TimeUnit::Second, None),
false,
),
]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(Int32Array::from(vec![1, 2, 3])),
Arc::new(StringArray::from(vec!["Alice", "Bob", "Charlie"])),
Arc::new(TimestampSecondArray::from(vec![
chrono::Utc::now().timestamp(),
chrono::Utc::now().timestamp(),
chrono::Utc::now().timestamp(),
])),
],
)
.unwrap();
let table = MemTable::try_new(schema, vec![vec![batch]]).unwrap();
ctx.register_table("users", Arc::new(table)).unwrap();
ctx
}
#[tokio::main]
async fn main() {
let ctx = create_test_session();
let sql = r#"
SELECT
CAST(
DATE_FORMAT(
CAST(
TO_DATE(
DATE_FORMAT(
CAST(`_modifiedDateTime` AS TIMESTAMP),
'%Y-%m-%dT%H:%i:%S'
),
'%Y-%m-%dT%H:%i:%S'
) AS TIMESTAMP
),
'%Y-%m-%d %H:%i:%s'
) AS TIMESTAMP
) AS qt_5anmnq5myd
FROM users group by qt_5anmnq5myd;
"#;
let plan = ctx.state().create_logical_plan(sql).await.unwrap();
plan.transform(|p| match p {
LogicalPlan::Projection(ref projection) => {
println!("DEBUG {:?}", projection.expr);
Ok(Transformed::no(p))
}
_ => Ok(Transformed::no(p)),
})
.unwrap();
}
```
### Suggested Behavior
The projection expression should remain a structured `Expr` tree regardless of whether a `GROUP BY` clause is present.
Specifically:
The logical plan should preserve the original composed expression (Cast, ScalarFunction, etc.).
Analyzer and optimizer rules should be able to traverse and rewrite the projection expression consistently for both grouped and non-grouped queries.
### Additional context
By tracing in datafusion/sql/src/select.rs, the expression is rewritten during aggregation planning:
```
if !group_by_exprs.is_empty() || !aggr_exprs.is_empty() {
self.aggregate(...)
}
```
Before aggregation: `select_exprs` contains the fully composed `Expr`.
After aggregation: `select_exprs_post_aggr` contains `Alias(Column())`, and the schema contains only this single stringified column.
This causes problems for downstream `AnalyzerRules`. In our case, we have an analyzer that rewrites schemas to match a remote database schema, but for grouped queries we can no longer identify or rewrite the original projection expression.
We implemented a workaround that attempts to reconstruct the projection expression by matching it back to the `GROUP BY` expression in the `AggregatePlan` and re-stringifying it, but this is fragile and not ideal.
A possible improvement would be to delay collapsing projection expressions into stringified Columns until later(e.g. after analyzer rules or during optimization planning), so expression structure is preserved during logical analysis.
CC: @backkem
Contributor guide
Research direction
Start in datafusion/sql/src/select.rs and trace the aggregate path reached when GROUP BY is present, comparing select_exprs with select_exprs_post_aggr. Reproduce the query through SessionContext::create_logical_plan and inspect the logical plan. Done means grouped projections preserve their structured Expr trees so analyzer and optimizer rules can traverse them consistently with non-grouped projections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100