apache / apache/datafusion

Decimal32/Decimal64 support in Substrait Producer

Open
#21,779 1 comment 0 reactions 1 assignee Claimed by @akoshchiy View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`datafusion-substrait`'s producer does not support Arrow `Decimal32` or `Decimal64`.
Any DataFusion plan with a `Decimal32`/`Decimal64` field or literal causes
`to_substrait_plan` to fail with:

`NotImplemented("Unsupported cast type: ")`
`NotImplemented("Unsupported literal: Decimal32(..)")`

### Describe the solution you'd like

Substrait has one abstract `Decimal { precision, scale }` and distinguishes
physical widths via `type_variation_reference`, same mechanism already used for
`Decimal256`. So:
1. Add `DECIMAL_32_TYPE_VARIATION_REF` / `DECIMAL_64_TYPE_VARIATION_REF` in
`variation_const.rs`, plus matching entries in Arrow's
`format/substrait/extension_types.yaml`.
2. Add `Decimal32` / `Decimal64` arms in `to_substrait_type_from_field` and
`to_substrait_literal`, mirroring the existing `Decimal128` arms.
3. Extend `round_trip_types` and `round_trip_literals` with `Decimal32(9, 2)`
and `Decimal64(18, 2)` cases.
4. [optionally] Make `consumer/types.rs::from_substrait_type` (and the literal consumer)
branch on `type_variation_reference` to pick the narrow Arrow width, falling
back to precision when the anchor is absent.

### Describe alternatives you've considered

None

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.