apache / apache/hudi

support reload timeline Incrementally

Open
#14,996 0 comments 0 reactions 0 assignees View on GitHub
area:core area:incr-processing area:writer from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Recently, call `HoodieTableMetaClient.reloadActiveTimeline` many times in one operation, and this will reload the timeline fully.

Perhaps, to support to reload in Incremental mode will increase the performance.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3232
- Type: Improvement

---

## Comments

23/Jan/22 19:40;shivnarayan;this definitely looks beneficial. 

may I know how you plan to get this done. 

Do you plan to add some filtering in below method in HoodieTableMetaClient. 
{code:java}
public List scanHoodieInstantsFromFileSystem(Path timelinePath, Set includedExtensions,
boolean applyLayoutVersionFilters) throws IOException { {code}
 

something like,

HTMC (HoodieTableMetaClient) will maintain last loaded instant or mod time of last loaded instant file. So, every time when reload is invoked, we just filter for files modified after the last known mod time. 

 

Or you had something else in mind ? 

 ;;;

---

25/Jan/22 07:38;biyan900116@gmail.com;[~shivnarayan]

i plan to do something in HoodieActiveTimeline.

all the methods in HoodieActiveTimeline will judge whether need to load the latest instant. If yes, load incrementally and execute the following codes.

Also, all the methods that will create a new instant will force to load after the operation.

WDYT?;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HoodieTableMetaClient.reloadActiveTimeline and HoodieActiveTimeline, then inspect scanHoodieInstantsFromFileSystem and the methods that create new instants. Confirm the intended reload behavior and identify existing timeline tests before changing anything; done means repeated reloads can discover new instants incrementally while newly created instants are still visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.