apache / apache/hudi

Deduce archival config based on hours derived from cleaner hours based config

Open
#15,973 0 comments 0 reactions 0 assignees View on GitHub
area:table-service component:archiving component:cleaning from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

We have hours based policy/configs for cleaning, but archival only has num_commits based configs. So, as of now, user/dev has to ensure they both align well together. We should derive archival configs automatically from cleaner configs. 

 

for eg, 

if cleaner configs is 48 hours, 

we can add 10% more for archival min commits and 20% for max commits. 

So, archival window can be 53 hours -> 58 hours. we can expose configs for these percentage if need be. 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6252
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating Hudi's cleaner and archival configuration definitions and review the requirements in JIRA HUDI-6252. Determine how hour-based cleaner settings relate to the existing commit-based archival settings, then define tests that verify the derived archival window and configurable percentages. Done means the settings stay aligned automatically and the behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.