[Parquet] Encoding configuration should be easier and more automated
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Right now, configuring good encoding values is difficult for users. There is nothing to help them make those decisions, and the defaults are a bit simplistic (try RLE_DICTIONARY then fall back on PLAIN, IIUC). If they want to override encodings, they have to do so on a column-by-column basis (which probably becomes very cumbersome if there hundreds of columns).
Ideally, there should be a way for users to get an automatic selection of encodings, based on their data or at least their data types (and also the selected Parquet version), that provides a good compromise between disk footprint and decoding speed.
(in Python, think `pq.write_table(..., column_encoding="auto")`)
Perhaps it would be also nice for users to pass per-datatype preferences, rather than per-column.
### Component(s)
C++, Parquet
Contributor guide
Assessment
This issue has not been assessed yet.