Introduce Hybrid Cleaner policy based on both LATEST_COMMITS and LATEST_FILE_VERSIONS
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
We have two major cleaner policies. LATEST_COMMITS and LATEST_FILE_VERSIONS. among this, LATEST_COMMITS might be [[efficient|https://github.com/apache/hudi/blob/570989dc4011d5370f60991d8782408dd0f72c07/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java#L177]], bcoz, we maintain earliest retained commit and will read into new commits (commit metadata) after the earliest retained to find the partitions that might be eligible for cleaning.
with LATEST_FILE_VERSIONS, we can't do any such optimization. So, we always end up doing [full listing|#L207].]
As you can imagine, for larger tables w/ huge no of partitions, this might have a hit on the perf. So, wondering if we can introduce a hybrid cleaner policy which combines both.
For eg, we will do cleaning based on LATEST_FILE_VERSIONS until Nth commit(say 10). And every 10th commit, we will trigger cleaner based on LATEST_COMMITS. so, that from 11th commit until 20th, we can do poll commits after 10 to get the list of partitions to clean instead of doing a full listing.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4750
- Type: Improvement
---
## Comments
03/Sep/22 14:02;zouxxyy;Maybe we can directly increase the default value of `hoodie.clean.max.commits` ? Currently it's 1, which I feel is a bit too fast;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java, especially the referenced LATEST_COMMITS and LATEST_FILE_VERSIONS paths. Read how hoodie.clean.max.commits affects cleaning, then trace the existing policy configuration and tests if present. Done means a hybrid policy can use file versions for the configured interval and commit metadata on each interval boundary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100