apache / apache/arrow-rs

[DISCUSS] Consider enabling block compression by default for parquet

Open
#7,407 1 comment 0 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

When using block compression, there is a tradeoff:
1. Smaller file sizes (and thus potentially more efficient file IO)
2. Longer decoding time (requires more CPU to decode the pages)

Most systems I know of in practice (e.g. duckdb, datafusion, influxdb 3.0) default to using page level compression, but the parquet defaults to no compression ([source here](https://docs.rs/parquet/latest/src/parquet/file/properties.rs.html#34))

@XiangpengHao suggests in https://github.com/apache/arrow-rs/issues/7363#issuecomment-2797292029

> As a side note, I think we should by default enable compression in parquet writer settings. As parquet doesn't have good string encodings, without block compressions, string columns are practically almost uncompressed.

**Describe the solution you'd like**
Enable compression by default

**Describe alternatives you've considered**

One question is if we should use default compressions for strings and non strings

1. I suggest we follow DuckDB's lead and default to `SNAPPY` compression to balance speed and compression ratio.
2. We could also use `ZSTD`, what DataFusion uses -- that gives higher compression ratios but slower performance
3. Don't change the default but better document the

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by reviewing parquet/file/properties.rs, the current writer defaults linked in the issue, and the referenced Arrow discussion. Compare the proposed SNAPPY and ZSTD choices, including separate string and non-string behavior, and define the expected default before making a change; the issue is done when the selected default is implemented or the documentation alternative is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.