apache / apache/arrow-rs

[parquet] Avoid materializing native primitive array when column's arrow type is Dictionary

Open
#8,250 0 comments 0 reactions 1 assignee Claimed by @Jefffrey View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When reading columns where the Arrow data type is a dictionary encoded `PrimitiveArray`, it seems like we're doing something a bit inefficient. We use the builder chooses to use a `PrimitiveArrayReader` here:
https://github.com/apache/arrow-rs/blob/3dcd23ffa3cbc0d9496e1660c6f68ce563a336b4/parquet/src/arrow/array_reader/primitive_array.rs#L482

Which will decode the native array, and then cast it back to a dictionary here:
https://github.com/apache/arrow-rs/blob/main/parquet/src/arrow/array_reader/primitive_array.rs

This seems inefficient because a lot of the values in this array get thrown away after the cast.

**Describe the solution you'd like**
It seems like it would be more efficient to decode directly into a dictionary array, similar to what we do dictionaries with byte array types. E.g.
- implement a `PrimitiveArrayDictionaryReader` implementation of `ArrayReader`, similar to what we've done for byte arrays here https://github.com/apache/arrow-rs/blob/main/parquet/src/arrow/array_reader/byte_array_dictionary.rs
- Choose the appropriate reader implementation in `ArrayRaederBuilder::build_primitiveReader` here https://github.com/apache/arrow-rs/blob/3dcd23ffa3cbc0d9496e1660c6f68ce563a336b4/parquet/src/arrow/array_reader/builder.rs#L303

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.