Make metasync to multiple catalogs multithreaded
Open
component:catalog-sync
from-jira
priority:medium
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
If you have multiple catalogs, metasync to each one happens sequentially. If a user needs this to be multithreaded it can be implemented pretty easily. [https://github.com/apache/hudi/pull/7318] is an example of how to do it
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5335
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the metasync implementation and reviewing the linked pull request 7318 for the existing multi-catalog approach. Run the relevant metasync tests, then verify that synchronization for multiple catalogs can proceed concurrently while preserving successful completion and error reporting for each catalog.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100