Enhance 8 to 7 Downgrade handler to avoid bringing entire byte[] to driver memory
Open
area:writer
from-jira
priority:critical
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
[https://github.com/apache/hudi/blob/720e7eb9e7107202b67b261fc20772d3a6394b7d/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java#L222]
We are reading entire record from LSM timeline as GenericRecord and creating equivalent archived metaEntry. But if we have a large commit metadata along w/ a large compaction plan, this might potentially cause OOM in driver.
Not sure if there is a easier way out since the record is represented as GenericRecord. but filing a tracking ticket.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9271
- Type: Improvement
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.