parquet: support set preferred encoding method for types
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently we can set the encoding method for the entire writer (using [`set_encoding()`](https://docs.rs/parquet/34.0.0/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_encoding)) or for each column (using [`set_column_encoding()`](https://docs.rs/parquet/34.0.0/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_column_encoding)). I'd like to propose another way that allows users to set preferred encoding method for each type. E.g., prefer PLAIN for float and DELTA_BINARY_PACKED for int64 etc.
**Describe the solution you'd like**
Add a new property that describes the preferred mapping. It can be something like
```rust
struct TypeEncoding {
boolean: Option,
int32: Option,
// ...
}
```
For priority among all those ways, we can let the more concrete setting overwrites the more general one. I.e, column encoding properties > type encoding properties > global encoding.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start by reading the linked WriterPropertiesBuilder methods set_encoding() and set_column_encoding() in the parquet properties API. Trace how global and column encoding settings are represented and applied, then determine where a per-type preference mapping would fit. Done means type-level preferences work with the proposed precedence: column encoding over type encoding over global encoding, with coverage for the supported types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100