Auto Compute Compression ratio for input data to output parquet/orc file size
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
Context :
Submitted a spark job to read 3-4B ORC records and wrote to Hudi format. Creating the following table with all the runs that I had carried out based on different options
||CONFIG ||Number of Files Created||Size of each file||
|PARQUET_FILE_MAX_BYTES=DEFAULT|30K|21MB|
|PARQUET_FILE_MAX_BYTES=1GB|3700|178MB|
|PARQUET_FILE_MAX_BYTES=1GB
COPY_ON_WRITE_TABLE_INSERT_SPLIT_SIZE=1100000|Same as before|Same as before|
|PARQUET_FILE_MAX_BYTES=1GB
BULKINSERT_PARALLELISM=100|Same as before|Same as before|
|PARQUET_FILE_MAX_BYTES=4GB|1600|675MB|
|PARQUET_FILE_MAX_BYTES=6GB|669|1012MB|
Based on this runs, it feels that the compression ratio is off.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-2003
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
- Fix version(s):
- 1.1.0
---
## Comments
14/Jun/21 12:31;vinaypatil18;[~nishith29] Please do update the description if I have missed anything here;;;
---
28/Feb/22 22:48;xushiyan;[~x1q1j1] more info in the related ticket HUDI-64;;;
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.