apache / apache/hudi

Introduce Hybrid Cleaner policy based on both LATEST_COMMITS and LATEST_FILE_VERSIONS

Open
#15,400 0 comments 0 reactions 0 assignees View on GitHub
component:cleaning from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

We have two major cleaner policies. LATEST_COMMITS and LATEST_FILE_VERSIONS. among this, LATEST_COMMITS might be [[efficient|https://github.com/apache/hudi/blob/570989dc4011d5370f60991d8782408dd0f72c07/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java#L177]], bcoz, we maintain earliest retained commit and will read into new commits (commit metadata) after the earliest retained to find the partitions that might be eligible for cleaning. 

with LATEST_FILE_VERSIONS, we can't do any such optimization. So, we always end up doing [full listing|#L207].]

 

As you can imagine, for larger tables w/ huge no of partitions, this might have a hit on the perf. So, wondering if we can introduce a hybrid cleaner policy which combines both. 

For eg, we will do cleaning based on LATEST_FILE_VERSIONS until Nth commit(say 10). And every 10th commit, we will trigger cleaner based on LATEST_COMMITS. so, that from 11th commit until 20th, we can do poll commits after 10 to get the list of partitions to clean instead of doing a full listing. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4750
- Type: Improvement

---

## Comments

03/Sep/22 14:02;zouxxyy;Maybe we can directly increase the default value of `hoodie.clean.max.commits` ? Currently it's 1, which I feel is a bit too fast;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java, especially the referenced LATEST_COMMITS and LATEST_FILE_VERSIONS paths. Read how hoodie.clean.max.commits affects cleaning, then trace the existing policy configuration and tests if present. Done means a hybrid policy can use file versions for the configured interval and commit metadata on each interval boundary.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.