Decimal32/Decimal64 support in Substrait Producer
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
`datafusion-substrait`'s producer does not support Arrow `Decimal32` or `Decimal64`.
Any DataFusion plan with a `Decimal32`/`Decimal64` field or literal causes
`to_substrait_plan` to fail with:
`NotImplemented("Unsupported cast type: ")`
`NotImplemented("Unsupported literal: Decimal32(..)")`
### Describe the solution you'd like
Substrait has one abstract `Decimal { precision, scale }` and distinguishes
physical widths via `type_variation_reference`, same mechanism already used for
`Decimal256`. So:
1. Add `DECIMAL_32_TYPE_VARIATION_REF` / `DECIMAL_64_TYPE_VARIATION_REF` in
`variation_const.rs`, plus matching entries in Arrow's
`format/substrait/extension_types.yaml`.
2. Add `Decimal32` / `Decimal64` arms in `to_substrait_type_from_field` and
`to_substrait_literal`, mirroring the existing `Decimal128` arms.
3. Extend `round_trip_types` and `round_trip_literals` with `Decimal32(9, 2)`
and `Decimal64(18, 2)` cases.
4. [optionally] Make `consumer/types.rs::from_substrait_type` (and the literal consumer)
branch on `type_variation_reference` to pick the narrow Arrow width, falling
back to precision when the anchor is absent.
### Describe alternatives you've considered
None
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.