allow ExternalSpillMap use accurate payload size rather than estimated
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Situation: In ExternalSpillMap, we need to control the amount of data in memory map to avoid OOM. Currently, we evaluate this by estimate the average size of each payload twice. And get total memory use by multiplying average payload size with payload number. The first time we get the size is when first payload is inserted while the second time is when there are 100 payloads stored in memory.
Problem: If the size is underestimated in the second estimation, an OOM will happen.
Plan: Could we have a flag to control if we want an evaluation in accurate?
Currently, I have several ideas but not sure which one could be the best or if there are any better one.
# Estimate each payload, store the length of payload with its value. Once update or remove happen, use diff old length and add new length if needed so that we keep the sum of all payload size precisely. This is the method I currently use in prod.
# Do not store the length but evaluate old payload again when it is popped. It trades off space against time comparing to method one. A better performance may be reached when updating and removing are rare. I didn't adopt this because I had profile ingestion process by arthas and found size estimating in that may be time consuming in flame graph. But I'm not sure whether it is true in compaction. In my intuition,HoodieRecordPayload has a quite simple structure.
# I also have a more accurate estimate method that is evaluate the whole map when size is 1,100,10000 and one million. Less underestimate will happen in such large amount of data.
Look forward to any advice or suggestion or discussion.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1796
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the ExternalSpillMap implementation and how it handles HoodieRecordPayload values. Compare the three payload-size accounting approaches described in the issue, including their memory and runtime tradeoffs. Done means an agreed approach accurately controls in-memory data volume without risking underestimation, with behavior validated against the HUDI-1796 requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100