Substrait decimal arithmetic derives return types that differ from the referenced functions
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
The Substrait consumer maps standard decimal arithmetic to DataFusion operators whose derived precision and scale can differ from the referenced Substrait function definition. It accepts the plan and exposes the different type without reporting the mismatch.
Reproduced on `main` at `1b6dc92a221b41088afa2753fe5526be6d057b7b`.
Each plan references `extension:io.substrait:functions_arithmetic_decimal` and declares the return type given by its YAML formula. The add, multiply and divide definitions are identical in spec v0.87.0, v0.102.0 and v0.103.0.
| Expression | Spec / declared output_type | DataFusion logical schema |
| --- | --- | --- |
| dec(10,2) + dec(5,1) | dec(11,2) | dec(11,2) |
| dec(38,10) + dec(38,10) | dec(38,9) | dec(38,10) |
| dec(10,2) * dec(5,1) | dec(16,3) | dec(16,3) |
| dec(38,10) * dec(38,10) | dec(38,6) | dec(38,20) |
| dec(10,2) / dec(5,1) | dec(21,8) | dec(15,6) |
Inputs and declared outputs are required. The table focuses on precision and scale, with the two small add/multiply cases serving as controls.
### To reproduce
From a DataFusion checkout:
```sh
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0
mkdir -p conformance-inputs datafusion/substrait/examples
cp conformance-cases/derived-schema/decimal_*.json conformance-inputs/
cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs
SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probe
```
The [probe](https://github.com/alexandrefimov/substrait-conformance-cases/blob/f12ba6103217fbd50b95c407b5dc3e8fae1e05a0/probe/datafusion_corpus_probe.rs) registers a table with the exact declared input types. Its `DATAFUSION ACCEPTED` line reports the logical schema before execution. For example, [`decimal_divide`](https://github.com/alexandrefimov/substrait-conformance-cases/blob/f12ba6103217fbd50b95c407b5dc3e8fae1e05a0/derived-schema/decimal_divide.json) declares `decimal(21,8)` and reports `Decimal128(15, 6)`.
### Expected behavior
The consumed expression should follow the return-type rule of the referenced [decimal function](https://github.com/substrait-io/substrait/blob/v0.103.0/extensions/functions_arithmetic_decimal.yaml), or reject a function contract it cannot implement. For division here, the YAML gives `scale = max(6, 2 + 5 + 1) = 8` and `precision = 10 - 2 + 5 + 8 = 21`.
`from_scalar_function` resolves the function name to a native operator and constructs a `BinaryExpr` without checking its return type against the declaration. This is a Substrait conversion issue, not a request to change the semantics of ordinary DataFusion SQL arithmetic.
The wide-add fixture also reports an execution overflow at scale 10. The core reproducer is the schema mismatch: these plans omit overflow options, so this report does not prescribe a rounding or overflow policy. In particular, the wide-multiply input value can overflow even with the correct declared type.
Contributor guide
Assessment
This issue has not been assessed yet.