apache / apache/hudi

[DISCUSSION] Should we change default compression codec for Parquet files to snappy?

Open
#13,568 7 comments 0 reactions 0 assignees View on GitHub
area:config type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Now the default value of `hoodie.parquet.compression.codec` (HoodieStorageConfig.PARQUET_COMPRESSION_CODEC_NAME) is `gzip`.
Recently I profiled compaction without configuring any codec and found that method `MessageColumnIORecordConsumer.endMessage()` (which makes compression, `gzip` by default) took **39%** (3,813,420 ms) of the whole time of writing. When run compaction with `snappy` on the same data, the compression took only **3.76%** (242,680 ms) of writing. So, snappy compression is >10 times faster. Why don't we use `snappy` as a default compression codec for parquet files?

If you agree with this proposal, i'm ready to make this change.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HoodieStorageConfig.PARQUET_COMPRESSION_CODEC_NAME and trace where the Parquet compression default is defined and tested. Review the reported MessageColumnIORecordConsumer.endMessage() profiling context and the discussion before deciding whether the default should change; done means a maintainer-approved codec choice with compatibility implications and coverage for the resulting default.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.