Substrait producer omits the required output_type on aggregate functions
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
The Substrait producer emits aggregate calls with `output_type: None`. The logical plan already has a known result type, but the exported call does not carry it. This affects `count`, `sum`, `avg` and `min` in the reproducer below.
Reproduced on main at `8a9228164fddeaa03c9ee510b641990e403f2eb5`.
### To reproduce
From a DataFusion checkout, using its pinned Rust toolchain:
```sh
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout c4d3ffd74842aa38079b3d523913199dc615a352
mkdir -p datafusion/substrait/examples
cp conformance-cases/probe/datafusion_producer_probe.rs datafusion/substrait/examples/corpus_producer.rs
cargo run --locked -p datafusion-substrait --example corpus_producer -- --aggregate-output-types
```
Use an unused example filename if `corpus_producer.rs` already exists. The probe registers an empty named table with a required Int64 column and calls `to_substrait_plan` for each aggregate. It inspects the produced protobuf directly, without converting it back through a consumer.
| Query | Logical output type | Exported AggregateFunction.output_type |
| --- | --- | --- |
| `SELECT count(i) FROM t` | Int64 | None |
| `SELECT sum(i) FROM t` | Int64 | None |
| `SELECT avg(i) FROM t` | Float64 | None |
| `SELECT min(i) FROM t` | Int64 | None |
The final diagnostic is `{"cases":4,"missing_output_types":4}`.
### Expected behavior
[`AggregateFunction.output_type`](https://github.com/substrait-io/substrait/blob/v0.87.0/proto/substrait/algebra.proto#L1806) must be set to the return type derived from the referenced function declaration. The producer should write a conforming type, or report that it cannot represent that function contract.
[`from_aggregate_function`](https://github.com/apache/datafusion/blob/8a9228164fddeaa03c9ee510b641990e403f2eb5/datafusion/substrait/src/logical_plan/producer/expr/aggregate_function.rs) currently sets `output_type: None` explicitly. A consumer that validates required fields can reject these plans; one that uses the declaration for schema inference has no type to use.
This report is about the missing declaration. It does not prescribe an integer AVG return type or address aggregate phases. #24967 and #25045 concern phase handling in the consumer; #15831 fixed the corresponding missing-output-type issue for BinaryExpr.
Contributor guide
Assessment
This issue has not been assessed yet.