Unbound the restriction that clean retain commits must be smaller than archive minimum commits
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The end-to-end streaming processing is more and more popular around the Flink users now, and the most typical application scenario for streaming ingestion checkpoint interval is within minutes (1min, 5mins ..). Say user sets up the time-interval as 1 minute, and there are about 60 write commits on the timeline for one hour.
{t1, t2, t3, t4 ...t60}
Now let's consider the very popular streaming read scenario, people want to keep the history data for a medium live time(usually 1 day or even 1 week), and let's say user configure the cleaning retain commits number as:
_1(day) * 24 (hours) * 60 (commits of one hour) _= *1440 commits*
While considering the current cleaning retain commits restriction:
_num_retain_commits < min_archive_commits_num_
We must keep at least 1440 commits on the active timeline, that means we have at least:
_1440 * 3 = 4320_
files on the timeline !!! Which is a pressure to the file IO and the metadata scanning (the metadata client). If we do not configure long enough retain time commits, the writer may remove the old files and the reader encounter {{FileNotFoundException}}.
So, we may find a way to lift restriction that active timeline commits number must be greater than cleaning retain commits.
One way i can think of is that we remember the last committed cleaning instant and only check that when cleaning (suitable for the hours cleaning strategy). With num_commits cleaning strategy we may need to scan the archive timeline (or metadata table if it is enabled ?)
Whatever a solution is eagerly needed now !
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3657
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the cleaning logic for the active and archive timelines, focusing on the num_commits and hours cleaning strategies and the min_archive_commits_num restriction. Check how the last committed cleaning instant and metadata table are represented, then define tests showing that readers retain the requested history without excessive active-timeline files or FileNotFoundException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100