apache / apache/arrow-rs

[Parquet] Support per-column encodings in parquet-rewrite

Open
#10,643 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

`parquet-rewrite --encoding` currently applies one encoding to every column. This makes type-specific encodings difficult to use with mixed schemas.

For example, ALP supports only `FLOAT` and `DOUBLE`. Applying it globally to a file containing other physical types fails after output-file creation. Other type-specific encodings have similar behavior.

This depends on #9372 and should be worked on after that PR is merged.

### Describe the solution you would like

Add per-column encoding selection to `parquet-rewrite`, while retaining the global encoding option as a default.

Validate the selected encoding against each column physical type before rewriting begins. Unsupported combinations should return an actionable error, for example that ALP supports only `FLOAT` and `DOUBLE`, without leaving an incomplete output file.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the parquet-rewrite implementation and the changes from dependent issue #9372. Trace how --encoding is parsed and applied across columns, then define the per-column selection and pre-rewrite physical-type validation. Done means mixed-schema files can use column-specific encodings, the global option remains the default, and unsupported combinations fail before an incomplete output file is created.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.