apache / apache/arrow-rs

Remove dictionary type from Arrow logical type

Open
#4,729 18 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

We've seen a lot of issues related to dictionary array support in both `arrow-rs` and DataFusion. One of main reasons, I think, is that `arrow-rs` treats dictionary type as part of its [logical type](https://github.com/apache/arrow-rs/blob/master/arrow-schema/src/datatype.rs#L40).

A better approach IMO is to consider dictionary type as a physical type property and hide it from the logical `DataType`. Correspondingly, `arrow-rs` shouldn't maintain separate `Int32Array` and `Int32DictionaryArray`, etc, but rather unifying the two and hide the encoding details inside the array implementation.

**Describe the solution you'd like**

- Remove `Dictionary` from Arrow `DataType`.
- Unify dictionary array with plain array implementation.

**Describe alternatives you've considered**

Not doing it, and live with the complexities.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with arrow-schema/src/datatype.rs around the linked DataType definition, then trace dictionary array support in arrow-rs and its interactions with DataFusion. Done means removing Dictionary from DataType and unifying dictionary and plain array implementations without preserving separate dictionary array types.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.