apache / apache/hudi

Long standing replace inflights post completion can be archived before cleaning for KEEP_LATEST_FILE_VERSIONS cleaner policy.

Open
#16,012 0 comments 0 reactions 0 assignees View on GitHub
area:table-service component:cleaning from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

For KEEP_LATEST_FILE_VERSIONS cleaner policy, file versions are only maintained for active file groups not for replaced file groups.

Since, earliestCommitToRetain is null for KEEP_LATEST_FILE_VERSIONS policy, last clean instant can be considered as a lower bound, since the cleaner would have removed all the file groups until then. But there is a catch to this logic, while cleaner is running if there is a pending replacecommit then those files are not cleaned.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6351
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Start by tracing cleaner handling for KEEP_LATEST_FILE_VERSIONS, especially how pending replace commits interact with earliestCommitToRetain and the last clean instant. Done means replaced file groups are not archived before their pending replace commits are cleaned.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.