[Parquet] Allow sampling some values to decide the encoding pattern
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Parquet V2 has rich encoding patterns, but configuring them hand by hande a bit hard. Maybe a way is like btrblocks, which can sampling some keys to decide the encoding pattern.
The most common pattern is whether to use dictionary encoding or delta encoding. This can be well done by sampling.
**Describe the solution you'd like**
Maybe a sampler? The DuckDB and Velox has similiar things.
**Describe alternatives you've considered**
Or a dictionary config:
1. Dictionary item bytes
2. Dictionary repeats in sampled rows, like 10000 non null rows should only have at most 5000 items when building dictionary. It's not perfect but in most cases it's useful.
**Additional context**
Related issues:
- https://github.com/apache/arrow-rs/issues/8358
- https://github.com/apache/arrow-rs/issues/9242
Contributor guide
Research direction
Start by reading the related Arrow Rust issues #8358 and #9242, then compare the sampling approaches mentioned from DuckDB and Velox. Done means the project has an agreed sampler or dictionary configuration that can choose between dictionary and delta encoding based on sampled values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100