Make compression adaptive with V2 data pages
Open
Good first issue
Type: enhancement
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
### Describe the enhancement requested
When writing a V2 data page, it seems that compression is always unconditionally enabled even when compression doesn't actually yield any benefits:
https://github.com/apache/parquet-java/blob/0fea3e1e22fffb0a25193e3efb9a5d090899458a/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java#L305-L311
It would be relatively easy to use a hardcoded threshold (for example 98%) above which compression is disabled, which makes reading faster.
### Component(s)
Core
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.