[DISCUSS] Consider enabling block compression by default for parquet
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When using block compression, there is a tradeoff:
1. Smaller file sizes (and thus potentially more efficient file IO)
2. Longer decoding time (requires more CPU to decode the pages)
Most systems I know of in practice (e.g. duckdb, datafusion, influxdb 3.0) default to using page level compression, but the parquet defaults to no compression ([source here](https://docs.rs/parquet/latest/src/parquet/file/properties.rs.html#34))
@XiangpengHao suggests in https://github.com/apache/arrow-rs/issues/7363#issuecomment-2797292029
> As a side note, I think we should by default enable compression in parquet writer settings. As parquet doesn't have good string encodings, without block compressions, string columns are practically almost uncompressed.
**Describe the solution you'd like**
Enable compression by default
**Describe alternatives you've considered**
One question is if we should use default compressions for strings and non strings
1. I suggest we follow DuckDB's lead and default to `SNAPPY` compression to balance speed and compression ratio.
2. We could also use `ZSTD`, what DataFusion uses -- that gives higher compression ratios but slower performance
3. Don't change the default but better document the
**Additional context**
Contributor guide
Research direction
Start by reviewing parquet/file/properties.rs, the current writer defaults linked in the issue, and the referenced Arrow discussion. Compare the proposed SNAPPY and ZSTD choices, including separate string and non-string behavior, and define the expected default before making a change; the issue is done when the selected default is implemented or the documentation alternative is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100