KEEP_LATEST_BY_HOURS should consider modified time instead of commit time while setting earliestCommitToRetain value
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In CleanPlanner, KEEP_LATEST_BY_HOURS is setting earliestCommitToRetain value by consider timestamp directly, this will introduce bug if there are out of order commits where commit with lower timestamp is completed much later than commits with higher timestamps.
This policy's implementation needs to be revisit.
It should basically store the timestamp until which it cleaned let this be t1. Next cleaner instant should consider all the partitions and files that are modified from the point of t1 until (currentime-x) hours. Whichever files are not valid they should be removed.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6352
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.