[Parquet] Support per-column encodings in parquet-rewrite
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Is your feature request related to a problem or challenge?
`parquet-rewrite --encoding` currently applies one encoding to every column. This makes type-specific encodings difficult to use with mixed schemas.
For example, ALP supports only `FLOAT` and `DOUBLE`. Applying it globally to a file containing other physical types fails after output-file creation. Other type-specific encodings have similar behavior.
This depends on #9372 and should be worked on after that PR is merged.
### Describe the solution you would like
Add per-column encoding selection to `parquet-rewrite`, while retaining the global encoding option as a default.
Validate the selected encoding against each column physical type before rewriting begins. Unsupported combinations should return an actionable error, for example that ALP supports only `FLOAT` and `DOUBLE`, without leaving an incomplete output file.
Contributor guide
Research direction
Start by reviewing the parquet-rewrite implementation and the changes from dependent issue #9372. Trace how --encoding is parsed and applied across columns, then define the per-column selection and pre-rewrite physical-type validation. Done means mixed-schema files can use column-specific encodings, the global option remains the default, and unsupported combinations fail before an incomplete output file is created.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100