Use Arrow Extension Types Instead of Type Variation Hacks
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
DataFusion encodes Arrow-specific types (like unsigned integers) by misusing [`type_variation_reference`](https://github.com/apache/datafusion/blob/e6fc5160312481f7df8da3d69321350f81238e78/datafusion/substrait/src/logical_plan/producer/types.rs#L68-L77). This violates Substrait's [technology principle](https://substrait.io/spec/technology_principles/) to avoid specialization for a single producer.
Per the [spec](https://substrait.io/types/type_variations/), `type_variation_reference` is for physical variations of the same type where "all variations are expected to have the same semantics." Signed and unsigned integers have different semantics.
Types affected:
- UInt8/16/32/64
- LargeUtf8/LargeBinary/LargeList
- Decimal256
- Duration
- Date64
- Time32
- Time64
## Solution
Use Arrow's official [extension_types.yaml](https://github.com/apache/arrow/blob/main/format/substrait/extension_types.yaml) which already defines these types (u8, u16, large_string, decimal256, etc.).
**Before:**
```
Type::I8 { type_variation_reference: 1 } // means UInt8
```
**After:**
```
extension_uris: [{ uri: ".../extension_types.yaml" }]
Type::UserDefined { name: "u8" }
```
The consumer already handles extension types, so backwards compatibility can be maintained.
Contributor guide
Research direction
Start in datafusion/substrait/src/logical_plan/producer/types.rs, especially the type-variation handling linked in the issue, and compare the affected Arrow types with Apache Arrow's extension_types.yaml. Then trace the consumer's existing extension-type handling and check how extension URIs are represented. Done means all listed types use the official extensions while preserving the stated backwards-compatibility behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100