apache / apache/hudi

Fix incremental cleaning for clean based on LATEST_FILE_VERSIONS

Open
#15,442 1 comment 0 reactions 0 assignees View on GitHub
component:cleaning from-jira priority:critical status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

clean based on LATEST_FILE_VERSIONS can be improved further since incremental clean is not enabled. lets see if we can improvise. 

 

context from author:

 

 

Currently incremental cleaning is run for both KEEP_LATEST_COMMITS, KEEP_LATEST_BY_HOURS
policies. It is not run when KEEP_LATEST_FILE_VERSIONS.

This can lead to not cleaning files. This PR fixes this problem by enabling incremental cleaning for KEEP_LATEST_FILE_VERSIONS only.

Here is the scenario of the problem:

Say we have 3 committed files in partition-A and we add a new commit in partition-B, and we trigger cleaning for the first time (full partition scan):
 {{partition-A/
commit-0.parquet
commit-1.parquet
commit-2.parquet
partition-B/
commit-3.parquet}}
In the case say we have KEEP_LATEST_COMMITS with CLEANER_COMMITS_RETAINED=3, the cleaner will remove the commit-0.parquet to keep 3 commits.
For the next cleaning, incremental cleaning will trigger, and won't consider partition-A/ until a new commit change it. In case no later commit changes partition-A then commit-1.parquet will stay forever. However it should be removed by the cleaner.

Now if in case of KEEP_LATEST_FILE_VERSIONS, the cleaner will only keep commit-2.parquet. Then it makes sense that incremental cleaning won't consider partition-A until it is changed. Because there is only one commit.

This is why incremental cleaning should only be enabled with KEEP_LATEST_FILE_VERSIONS

Hope this is clear enough

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4878
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named in the issue. Start by locating the cleaner handling for KEEP_LATEST_COMMITS, KEEP_LATEST_BY_HOURS, and KEEP_LATEST_FILE_VERSIONS, then verify that incremental cleaning is enabled only for KEEP_LATEST_FILE_VERSIONS and that the described partition scenario removes obsolete files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.