apache / apache/hudi

Auto Compute Compression ratio for input data to output parquet/orc file size

Open
#14,805 0 comments 0 reactions 1 assignee Claimed by @ad1happy2go View on GitHub
area:writer from-jira priority:medium type:community-support type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

Context : 

Submitted  a spark job to read 3-4B ORC records and wrote to Hudi format. Creating the following table with all the runs that I had carried out based on different options

 
||CONFIG ||Number of Files Created||Size of each file||
|PARQUET_FILE_MAX_BYTES=DEFAULT|30K|21MB|
|PARQUET_FILE_MAX_BYTES=1GB|3700|178MB|
|PARQUET_FILE_MAX_BYTES=1GB
COPY_ON_WRITE_TABLE_INSERT_SPLIT_SIZE=1100000|Same as before|Same as before|
|PARQUET_FILE_MAX_BYTES=1GB
BULKINSERT_PARALLELISM=100|Same as before|Same as before|
|PARQUET_FILE_MAX_BYTES=4GB|1600|675MB|
|PARQUET_FILE_MAX_BYTES=6GB|669|1012MB|

Based on this runs, it feels that the compression ratio is off. 

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-2003
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
- Fix version(s):
- 1.1.0

---

## Comments

14/Jun/21 12:31;vinaypatil18;[~nishith29] Please do update the description if I have missed anything here;;;

---

28/Feb/22 22:48;xushiyan;[~x1q1j1]  more info in the related ticket HUDI-64;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.