Fix clean planning for very large partitions
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
Within clean planning phase, we do a map() for every partition and then trigger planning for each partition within that.
For very large number of partitions, and if cleaner shuffle parallelism is small, this results in more sequential planning. We can enhance this with mapPartitions call and optimize it
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5012
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the clean planning phase and the per-partition map() calls described in the issue, then inspect how cleaner shuffle parallelism affects planning. Compare the existing behavior with the mapPartitions approach; done means very large partition counts no longer cause avoidable sequential planning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100