Deduce archival config based on hours derived from cleaner hours based config
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
We have hours based policy/configs for cleaning, but archival only has num_commits based configs. So, as of now, user/dev has to ensure they both align well together. We should derive archival configs automatically from cleaner configs.
for eg,
if cleaner configs is 48 hours,
we can add 10% more for archival min commits and 20% for max commits.
So, archival window can be 53 hours -> 58 hours. we can expose configs for these percentage if need be.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6252
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating Hudi's cleaner and archival configuration definitions and review the requirements in JIRA HUDI-6252. Determine how hour-based cleaner settings relate to the existing commit-based archival settings, then define tests that verify the derived archival window and configurable percentages. Done means the settings stay aligned automatically and the behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100