apache / apache/arrow

[Parquet] Encoding configuration should be easier and more automated

Open
#38,441 20 comments 2 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Right now, configuring good encoding values is difficult for users. There is nothing to help them make those decisions, and the defaults are a bit simplistic (try RLE_DICTIONARY then fall back on PLAIN, IIUC). If they want to override encodings, they have to do so on a column-by-column basis (which probably becomes very cumbersome if there hundreds of columns).

Ideally, there should be a way for users to get an automatic selection of encodings, based on their data or at least their data types (and also the selected Parquet version), that provides a good compromise between disk footprint and decoding speed.

(in Python, think `pq.write_table(..., column_encoding="auto")`)

Perhaps it would be also nice for users to pass per-datatype preferences, rather than per-column.

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.