apache / apache/datafusion

Substrait decimal arithmetic derives return types that differ from the referenced functions

Open
#25,043 1 comment 0 reactions 1 assignee Claimed by @Satyr09 View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

The Substrait consumer maps standard decimal arithmetic to DataFusion operators whose derived precision and scale can differ from the referenced Substrait function definition. It accepts the plan and exposes the different type without reporting the mismatch.

Reproduced on `main` at `1b6dc92a221b41088afa2753fe5526be6d057b7b`.

Each plan references `extension:io.substrait:functions_arithmetic_decimal` and declares the return type given by its YAML formula. The add, multiply and divide definitions are identical in spec v0.87.0, v0.102.0 and v0.103.0.

| Expression | Spec / declared output_type | DataFusion logical schema |
| --- | --- | --- |
| dec(10,2) + dec(5,1) | dec(11,2) | dec(11,2) |
| dec(38,10) + dec(38,10) | dec(38,9) | dec(38,10) |
| dec(10,2) * dec(5,1) | dec(16,3) | dec(16,3) |
| dec(38,10) * dec(38,10) | dec(38,6) | dec(38,20) |
| dec(10,2) / dec(5,1) | dec(21,8) | dec(15,6) |

Inputs and declared outputs are required. The table focuses on precision and scale, with the two small add/multiply cases serving as controls.

### To reproduce

From a DataFusion checkout:

```sh
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0
mkdir -p conformance-inputs datafusion/substrait/examples
cp conformance-cases/derived-schema/decimal_*.json conformance-inputs/
cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs
SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probe
```

The [probe](https://github.com/alexandrefimov/substrait-conformance-cases/blob/f12ba6103217fbd50b95c407b5dc3e8fae1e05a0/probe/datafusion_corpus_probe.rs) registers a table with the exact declared input types. Its `DATAFUSION ACCEPTED` line reports the logical schema before execution. For example, [`decimal_divide`](https://github.com/alexandrefimov/substrait-conformance-cases/blob/f12ba6103217fbd50b95c407b5dc3e8fae1e05a0/derived-schema/decimal_divide.json) declares `decimal(21,8)` and reports `Decimal128(15, 6)`.

### Expected behavior

The consumed expression should follow the return-type rule of the referenced [decimal function](https://github.com/substrait-io/substrait/blob/v0.103.0/extensions/functions_arithmetic_decimal.yaml), or reject a function contract it cannot implement. For division here, the YAML gives `scale = max(6, 2 + 5 + 1) = 8` and `precision = 10 - 2 + 5 + 8 = 21`.

`from_scalar_function` resolves the function name to a native operator and constructs a `BinaryExpr` without checking its return type against the declaration. This is a Substrait conversion issue, not a request to change the semantics of ordinary DataFusion SQL arithmetic.

The wide-add fixture also reports an execution overflow at scale 10. The core reproducer is the schema mismatch: these plans omit overflow options, so this report does not prescribe a rounding or overflow policy. In particular, the wide-multiply input value can overflow even with the correct declared type.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.