apache / apache/arrow-rs

Improve dictionary decoder

Open
#9,670 1 comment 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Dictionary decoder has some hot paths in decoding

**Describe the solution you'd like**

We can unroll the dictionary gather/scatter loop to vectorize / improve throughput in this loop.
Also we can improve the codegen for `BitReader::get_value` by not going through a `slice`

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the dictionary decoder and BitReader::get_value implementations, then inspect the dictionary gather/scatter loop and its existing benchmarks. Compare the current and revised code generation and throughput; done means the hot paths are measurably improved without changing decoding behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.