`ProjectionExec` produces unknown statistics for all `ScalarFunctionExpr` outputs
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
`ProjectionExec::project_statistics()` only propagates column statistics for plain `Column` references and `Literal` values. Any `ScalarFunctionExpr` like `get_field()`, `variant_get`, or any UDF produces `ColumnStatistics::new_unknown()`:
https://github.com/apache/datafusion/blob/bc2b36cf56846e0c697b0f8b98619f346a72a9bf/datafusion/physical-expr/src/projection.rs#L716-L719
This means every column produced by a scalar function has `Absent` min/max/null_count/distinct_count, even when the function is a pure extraction (`get_field`, `variant_get`), a monotonic transformation (`cast`, `abs`), or any UDF where output stats are derivable from input stats
### Describe the solution you'd like
Add an optional method to `ScalarUdfImpl`:
something like `fn output_statistics(&self, input_statistics: &[ColumnStatistics]) -> Option`
This way, `ProjectionExec::project_statistics` would call this before falling back to unknown
### How this affects Struct + Variant queries
This matters for struct/variant queries especially, most output columns come through `get_field` or equivalent UDFs, so the cost model is effectively blind!
- **cost based join ordering** can't estimate cardinality for join keys produced by udfs
- **aggregation planning** can't estimate group count for group by on udf outputs
- **filterexec selectivity** - can't narrow row estimates for filters on udf outputs
Here's an MRE: https://github.com/apache/datafusion/compare/main...pydantic:datafusion:stats-propagation-mre?expand=1
Contributor guide
Research direction
Start in datafusion/physical-expr/src/projection.rs at ProjectionExec::project_statistics(), then trace the ScalarUdfImpl interface used by ScalarFunctionExpr. Use the linked MRE to observe statistics for get_field(), variant_get, and other UDF outputs. Done means derivable output statistics are propagated while unsupported functions still fall back to unknown statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100