Enabling asynchronous processing of Metastore Sync
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, runMetaSync() invokes each Metastore Sync in a blocking fashion, and iterates over the different Metastores sequentially - ([code link|https://github.com/apache/hudi/blob/release-0.12.0/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L695] within 0.12.0 branch)
And runMetaSync() is invoked during each commit, which can lead to a slow down of commit flow if many metastores are being synced. So enable async invocation of runMetaSync().
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5009
- Type: Task
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java at runMetaSync(), referenced around line 695 in the release-0.12.0 branch. Trace how it is invoked during each commit and how metastores are processed; done means Metastore Sync invocations no longer block the commit flow or require sequential processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100