Remove dictionary type from Arrow logical type
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We've seen a lot of issues related to dictionary array support in both `arrow-rs` and DataFusion. One of main reasons, I think, is that `arrow-rs` treats dictionary type as part of its [logical type](https://github.com/apache/arrow-rs/blob/master/arrow-schema/src/datatype.rs#L40).
A better approach IMO is to consider dictionary type as a physical type property and hide it from the logical `DataType`. Correspondingly, `arrow-rs` shouldn't maintain separate `Int32Array` and `Int32DictionaryArray`, etc, but rather unifying the two and hide the encoding details inside the array implementation.
**Describe the solution you'd like**
- Remove `Dictionary` from Arrow `DataType`.
- Unify dictionary array with plain array implementation.
**Describe alternatives you've considered**
Not doing it, and live with the complexities.
**Additional context**
Contributor guide
Research direction
Start with arrow-schema/src/datatype.rs around the linked DataType definition, then trace dictionary array support in arrow-rs and its interactions with DataFusion. Done means removing Dictionary from DataType and unifying dictionary and plain array implementations without preserving separate dictionary array types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100