apache / apache/datafusion

Substrait producer omits the required output_type on aggregate functions

Open
#25,049 4 comments 0 reactions 1 assignee Claimed by @namanjain24-sudo View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

The Substrait producer emits aggregate calls with `output_type: None`. The logical plan already has a known result type, but the exported call does not carry it. This affects `count`, `sum`, `avg` and `min` in the reproducer below.

Reproduced on main at `8a9228164fddeaa03c9ee510b641990e403f2eb5`.

### To reproduce

From a DataFusion checkout, using its pinned Rust toolchain:

```sh
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout c4d3ffd74842aa38079b3d523913199dc615a352
mkdir -p datafusion/substrait/examples
cp conformance-cases/probe/datafusion_producer_probe.rs datafusion/substrait/examples/corpus_producer.rs
cargo run --locked -p datafusion-substrait --example corpus_producer -- --aggregate-output-types
```

Use an unused example filename if `corpus_producer.rs` already exists. The probe registers an empty named table with a required Int64 column and calls `to_substrait_plan` for each aggregate. It inspects the produced protobuf directly, without converting it back through a consumer.

| Query | Logical output type | Exported AggregateFunction.output_type |
| --- | --- | --- |
| `SELECT count(i) FROM t` | Int64 | None |
| `SELECT sum(i) FROM t` | Int64 | None |
| `SELECT avg(i) FROM t` | Float64 | None |
| `SELECT min(i) FROM t` | Int64 | None |

The final diagnostic is `{"cases":4,"missing_output_types":4}`.

### Expected behavior

[`AggregateFunction.output_type`](https://github.com/substrait-io/substrait/blob/v0.87.0/proto/substrait/algebra.proto#L1806) must be set to the return type derived from the referenced function declaration. The producer should write a conforming type, or report that it cannot represent that function contract.

[`from_aggregate_function`](https://github.com/apache/datafusion/blob/8a9228164fddeaa03c9ee510b641990e403f2eb5/datafusion/substrait/src/logical_plan/producer/expr/aggregate_function.rs) currently sets `output_type: None` explicitly. A consumer that validates required fields can reject these plans; one that uses the declaration for schema inference has no type to use.

This report is about the missing declaration. It does not prescribe an integer AVG return type or address aggregate phases. #24967 and #25045 concern phase handling in the consumer; #15831 fixed the corresponding missing-output-type issue for BinaryExpr.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.