apache / apache/datafusion

Use Arrow Extension Types Instead of Type Variation Hacks

Open
#19,944 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

DataFusion encodes Arrow-specific types (like unsigned integers) by misusing [`type_variation_reference`](https://github.com/apache/datafusion/blob/e6fc5160312481f7df8da3d69321350f81238e78/datafusion/substrait/src/logical_plan/producer/types.rs#L68-L77). This violates Substrait's [technology principle](https://substrait.io/spec/technology_principles/) to avoid specialization for a single producer.

Per the [spec](https://substrait.io/types/type_variations/), `type_variation_reference` is for physical variations of the same type where "all variations are expected to have the same semantics." Signed and unsigned integers have different semantics.

Types affected:
- UInt8/16/32/64
- LargeUtf8/LargeBinary/LargeList
- Decimal256
- Duration
- Date64
- Time32
- Time64

## Solution

Use Arrow's official [extension_types.yaml](https://github.com/apache/arrow/blob/main/format/substrait/extension_types.yaml) which already defines these types (u8, u16, large_string, decimal256, etc.).

**Before:**
```
Type::I8 { type_variation_reference: 1 } // means UInt8
```

**After:**
```
extension_uris: [{ uri: ".../extension_types.yaml" }]
Type::UserDefined { name: "u8" }
```

The consumer already handles extension types, so backwards compatibility can be maintained.

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/substrait/src/logical_plan/producer/types.rs, especially the type-variation handling linked in the issue, and compare the affected Arrow types with Apache Arrow's extension_types.yaml. Then trace the consumer's existing extension-type handling and check how extension URIs are represented. Done means all listed types use the official extensions while preserving the stated backwards-compatibility behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.