apache / apache/hudi

[SUPPORT] Avoid rescan file when clean

Open
#11,647 7 comments 0 reactions 0 assignees View on GitHub
area:table-service type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Dear community, when using the cow table, I found that it may trigger an OOM error in driver when clean. I find that this is due to cow table rarely updating data, so there are usually no files that need to be cleaned. However, the clean operation is still called every time data is written, and when the number of files in the list reaches a certain threshold, it may cause OOM.

Case like:
instant1 instant2 instant3 instant4(scan 1、2、3、4 but no clean)
instant1 instant2 instant3 instant4 instant5 (scan 1、2、3、4、5 but no clean)

Is it possible to add an empty clean instant to mark it, to avoid rescanning every time, or are there any better ideas? Looking forward to your reply~

**Environment Description**

* Hudi version :0.14.0

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names no files, tests, or entry points. Start by reproducing the repeated clean calls with a Hudi 0.14.0 COW table whose data rarely changes, then trace the clean operation and its file-list handling. Done should prevent unnecessary rescans during clean and avoid the reported driver OOM, with a regression test covering the no-files-to-clean case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.