[DISCUSSION] Should we change default compression codec for Parquet files to snappy?
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Now the default value of `hoodie.parquet.compression.codec` (HoodieStorageConfig.PARQUET_COMPRESSION_CODEC_NAME) is `gzip`.
Recently I profiled compaction without configuring any codec and found that method `MessageColumnIORecordConsumer.endMessage()` (which makes compression, `gzip` by default) took **39%** (3,813,420 ms) of the whole time of writing. When run compaction with `snappy` on the same data, the compression took only **3.76%** (242,680 ms) of writing. So, snappy compression is >10 times faster. Why don't we use `snappy` as a default compression codec for parquet files?
If you agree with this proposal, i'm ready to make this change.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HoodieStorageConfig.PARQUET_COMPRESSION_CODEC_NAME and trace where the Parquet compression default is defined and tested. Review the reported MessageColumnIORecordConsumer.endMessage() profiling context and the discussion before deciding whether the default should change; done means a maintainer-approved codec choice with compatibility implications and coverage for the resulting default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100