Long standing replace inflights post completion can be archived before cleaning for KEEP_LATEST_FILE_VERSIONS cleaner policy.
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
For KEEP_LATEST_FILE_VERSIONS cleaner policy, file versions are only maintained for active file groups not for replaced file groups.
Since, earliestCommitToRetain is null for KEEP_LATEST_FILE_VERSIONS policy, last clean instant can be considered as a lower bound, since the cleaner would have removed all the file groups until then. But there is a catch to this logic, while cleaner is running if there is a pending replacecommit then those files are not cleaned.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6351
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Start by tracing cleaner handling for KEEP_LATEST_FILE_VERSIONS, especially how pending replace commits interact with earliestCommitToRetain and the last clean instant. Done means replaced file groups are not archived before their pending replace commits are cleaned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100