apache / apache/arrow-rs

Separate `parquet::basic::Compression` into two enums

Open
#9,863 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The `Compression` enum in the parquet crate currently serves two purposes. First it is part of the Parquet thrift schema, and is used for informational purposes in the metadata. In this role, it is a stand-in for the actual Thrift `CompressionCodec`.

Its second purpose is as a configuration parameter to control the writing of Parquet pages. As such, it is desirable for it to carry extra information to fine tune the compression codecs in use. But as more information is added, the memory burden on the metadata increases.

**Describe the solution you'd like**
I propose splitting this enum into two. `Compression` can move to `file::properties` for use in configuration. A new `CompressionCodec` that can make use of the thrift macros would be created in `basic`.

**Describe alternatives you've considered**
To keep breaking changes to a minimum, `Compression` could be simply left in `basic`. Also `ColumnChunkMetaData::compression` and `ColumnChunkMetaDataBuilder::set_compression` could keep their current signatures and simply convert internally.

**Additional context**
This would be a breaking API change, but I think it make adding new features to the writer easier while simplifying the parsing and representation of the Parquet metadata.

I thought of this while evaluating #9807 and #9367

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.