apache / apache/arrow-rs

[Parquet] Allow sampling some values to decide the encoding pattern

Open
#8,378 13 comments 3 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Parquet V2 has rich encoding patterns, but configuring them hand by hande a bit hard. Maybe a way is like btrblocks, which can sampling some keys to decide the encoding pattern.

The most common pattern is whether to use dictionary encoding or delta encoding. This can be well done by sampling.

**Describe the solution you'd like**

Maybe a sampler? The DuckDB and Velox has similiar things.

**Describe alternatives you've considered**

Or a dictionary config:
1. Dictionary item bytes
2. Dictionary repeats in sampled rows, like 10000 non null rows should only have at most 5000 items when building dictionary. It's not perfect but in most cases it's useful.

**Additional context**

Related issues:
- https://github.com/apache/arrow-rs/issues/8358
- https://github.com/apache/arrow-rs/issues/9242

Contributor guide

Open the contributing guide

Research direction

Start by reading the related Arrow Rust issues #8358 and #9242, then compare the sampling approaches mentioned from DuckDB and Velox. Done means the project has an agreed sampler or dictionary configuration that can choose between dictionary and delta encoding based on sampled values.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.