Auto-tune dedup parallelism without affecting write parallelism
Open
from-jira
priority:high
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
See HUDI-6863 and https://github.com/apache/hudi/pull/6802#issuecomment-1455802492
I think we need to make sure that the dedup parallelism is only applied to the dedup stage, not affecting subsequent stages, which may require better parallelism control by repartitioning with right parallelism before workload profiling.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6864
- Type: Improvement
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.