apache / apache/arrow-rs

parquet: support set preferred encoding method for types

Open
#3,823 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently we can set the encoding method for the entire writer (using [`set_encoding()`](https://docs.rs/parquet/34.0.0/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_encoding)) or for each column (using [`set_column_encoding()`](https://docs.rs/parquet/34.0.0/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_column_encoding)). I'd like to propose another way that allows users to set preferred encoding method for each type. E.g., prefer PLAIN for float and DELTA_BINARY_PACKED for int64 etc.

**Describe the solution you'd like**

Add a new property that describes the preferred mapping. It can be something like
```rust
struct TypeEncoding {
boolean: Option,
int32: Option,
// ...
}
```

For priority among all those ways, we can let the more concrete setting overwrites the more general one. I.e, column encoding properties > type encoding properties > global encoding.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked WriterPropertiesBuilder methods set_encoding() and set_column_encoding() in the parquet properties API. Trace how global and column encoding settings are represented and applied, then determine where a per-type preference mapping would fit. Done means type-level preferences work with the proposed precedence: column encoding over type encoding over global encoding, with coverage for the supported types.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.