Optimize column stats storage
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Avoid storing filename of each record in the colstats partition.
As of now, we store fileName as part of value in Col stats entries. This results in more storage, but comes w/ ease of getting everything in 1 look up. But as you could see, file name is repeated in every entries' value. And since its UUID based, each file name is going to add 70 bytes to each entry.
For eg,
lets say we have a table with 1000 columns. 1000 partitions. with each partition having 10k files.
Total entries in col stats partition = 1000*1000*10000 = 10^10. 10B records. So, thats ~ 70GB.
where in, if we can come up with a mapping of a unique Id for every filename, and store the mapping elsewhere (like FILES partition), we need only 8 bytes per entry.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3777
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-1822
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how fileName is stored in Col stats entries and how the colstats and FILES partitions are read and written. Review the JIRA task for the intended mapping design and its lookup implications. Done means repeated filenames are removed from colstats values while filename resolution remains available through the FILES partition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100