support reload timeline Incrementally
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Recently, call `HoodieTableMetaClient.reloadActiveTimeline` many times in one operation, and this will reload the timeline fully.
Perhaps, to support to reload in Incremental mode will increase the performance.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3232
- Type: Improvement
---
## Comments
23/Jan/22 19:40;shivnarayan;this definitely looks beneficial.
may I know how you plan to get this done.
Do you plan to add some filtering in below method in HoodieTableMetaClient.
{code:java}
public List scanHoodieInstantsFromFileSystem(Path timelinePath, Set includedExtensions,
boolean applyLayoutVersionFilters) throws IOException { {code}
something like,
HTMC (HoodieTableMetaClient) will maintain last loaded instant or mod time of last loaded instant file. So, every time when reload is invoked, we just filter for files modified after the last known mod time.
Or you had something else in mind ?
;;;
---
25/Jan/22 07:38;biyan900116@gmail.com;[~shivnarayan]
i plan to do something in HoodieActiveTimeline.
all the methods in HoodieActiveTimeline will judge whether need to load the latest instant. If yes, load incrementally and execute the following codes.
Also, all the methods that will create a new instant will force to load after the operation.
WDYT?;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HoodieTableMetaClient.reloadActiveTimeline and HoodieActiveTimeline, then inspect scanHoodieInstantsFromFileSystem and the methods that create new instants. Confirm the intended reload behavior and identify existing timeline tests before changing anything; done means repeated reloads can discover new instants incrementally while newly created instants are still visible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100