apache / apache/arrow-rs

Override DeltaBitPackEncoder Defaults

Open
#2,282 1 comment 0 reactions 1 assignee Claimed by @emecii View on GitHub
enhancement help wanted
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

The DeltaBitPackEncoder has a couple of parameters that users may wish to override, in particular:

* Block size - the number of values per block (default: 128)
* Miniblock count - the number of miniblocks per block (default: 4)

These defaults result in 32 elements per miniblock, the smallest possible. Small miniblocks may lead to smaller representation as the bit width is per miniblock, similarly small blocks may lead to smaller representation as the min delta is per block. However, for input data with a uniform distribution, larger blocks may yield better compression and faster decode.

See [here](https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5) for more information on what these parameters control

**Describe the solution you'd like**

`ColumnProperties` currently allows setting `Encoding` for a specific column, we should provide the ability to control the settings passed to the DeltaBitPackEncoder.

**Describe alternatives you've considered**

**Additional context**

#2276 adds vectorised unpacking of runs of 64x 64-bit numbers, however, this would need miniblocks containing 64 elements to benefit from this

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.