apache / apache/hudi

Unbound the restriction that clean retain commits must be smaller than archive minimum commits

Open
#15,072 0 comments 0 reactions 0 assignees View on GitHub
area:core from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The end-to-end streaming processing is more and more popular around the Flink users now, and the most typical application scenario for streaming ingestion checkpoint interval is within minutes (1min, 5mins ..). Say user sets up the time-interval as 1 minute, and there are about 60 write commits on the timeline for one hour.

{t1, t2, t3, t4 ...t60}

Now let's consider the very popular streaming read scenario, people want to keep the history data for a medium live time(usually 1 day or even 1 week), and let's say user configure the cleaning retain commits number as:

_1(day) * 24 (hours) * 60 (commits of one hour) _= *1440 commits*

While considering the current cleaning retain commits restriction:

_num_retain_commits < min_archive_commits_num_

We must keep at least 1440 commits on the active timeline, that means we have at least:

_1440 * 3 = 4320_

files on the timeline !!! Which is a pressure to the file IO and the metadata scanning (the metadata client). If we do not configure long enough retain time commits, the writer may remove the old files and the reader encounter {{FileNotFoundException}}.

So, we may find a way to lift restriction that active timeline commits number must be greater than cleaning retain commits.

One way i can think of is that we remember the last committed cleaning instant and only check that when cleaning (suitable for the hours cleaning strategy). With num_commits cleaning strategy we may need to scan the archive timeline (or metadata table if it is enabled ?)

Whatever a solution is eagerly needed now !

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3657
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the cleaning logic for the active and archive timelines, focusing on the num_commits and hours cleaning strategies and the min_archive_commits_num restriction. Check how the last committed cleaning instant and metadata table are represented, then define tests showing that readers retain the requested history without excessive active-timeline files or FileNotFoundException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.