[SUPPORT] Avoid rescan file when clean
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Dear community, when using the cow table, I found that it may trigger an OOM error in driver when clean. I find that this is due to cow table rarely updating data, so there are usually no files that need to be cleaned. However, the clean operation is still called every time data is written, and when the number of files in the list reaches a certain threshold, it may cause OOM.
Case like:
instant1 instant2 instant3 instant4(scan 1、2、3、4 but no clean)
instant1 instant2 instant3 instant4 instant5 (scan 1、2、3、4、5 but no clean)
Is it possible to add an empty clean instant to mark it, to avoid rescanning every time, or are there any better ideas? Looking forward to your reply~
**Environment Description**
* Hudi version :0.14.0
Contributor guide
No contributing guide indexed for this repository
Research direction
The report names no files, tests, or entry points. Start by reproducing the repeated clean calls with a Hudi 0.14.0 COW table whose data rarely changes, then trace the clean operation and its file-list handling. Done should prevent unnecessary rescans during clean and avoid the reported driver OOM, with a regression test covering the no-files-to-clean case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100