Add ability to dynamically configure write parallelism for BULK_INSERT for HoodieStreamer
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Add ability to dynamically configure write parallelism for BULK_INSERT for HoodieStreamer. Currently, BULK_INSERT parallelism to configured based on source parallelism that may be aggressive or conservative depending on other factors, e.g. partitions written to etc.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7855
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how HoodieStreamer currently determines BULK_INSERT parallelism from source parallelism, then review the linked JIRA issue for the intended configuration behavior. Done means the write parallelism can be configured dynamically and the setting is applied to BULK_INSERT instead of relying only on source parallelism.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100