apache / apache/hudi

Optimize column stats storage

Open
#15,100 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:critical type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Avoid storing filename of each record in the colstats partition.

As of now, we store fileName as part of value in Col stats entries. This results in more storage, but comes w/ ease of getting everything in 1 look up. But as you could see, file name is repeated in every entries' value. And since its UUID based, each file name is going to add 70 bytes to each entry.

For eg,
lets say we have a table with 1000 columns. 1000 partitions. with each partition having 10k files.

Total entries in col stats partition = 1000*1000*10000 = 10^10. 10B records. So, thats ~ 70GB.

where in, if we can come up with a mapping of a unique Id for every filename, and store the mapping elsewhere (like FILES partition), we need only 8 bytes per entry.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3777
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-1822

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how fileName is stored in Col stats entries and how the colstats and FILES partitions are read and written. Review the JIRA task for the intended mapping design and its lookup implications. Done means repeated filenames are removed from colstats values while filename resolution remains available through the FILES partition.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.