apache / apache/arrow-rs

Vectorised Pack Bits within BitWriter

Open
#2,257 1 comment 0 reactions 1 assignee Claimed by @sdf-jkl View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

`BitReader` has a vectorised `get_batch` method for unpacking blocks of bit-packed values. This provides a significant performance improvement over processing each value separately.

`BitWriter`, however, has no corresponding functionality to write blocks of values, and this represents a major bottleneck for encoding primitive values.

**Describe the solution you'd like**

We should add a `BitWriter::put_batch` method that can vectorise packing runs of primitives

**Describe alternatives you've considered**

We could not do this

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.