Use proper parallelism for engine context APIs
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
do a global search of these APIs
- org.apache.hudi.common.engine.HoodieEngineContext#flatMap
- org.apache.hudi.common.engine.HoodieEngineContext#map
and similar ones take in parallelism.
A lot of occurrences are using number of items as parallelism, which affect performance. Parallelism should be based on num cores available in the cluster and set by user via parallelism configs.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5261
- Type: Improvement
---
## Comments
08/Dec/22 23:47;jonvex;I see in this guide [https://spark.apache.org/docs/3.2.1/rdd-programming-guide.html#resilient-distributed-datasets-rdds] that it suggests 2-4 per core in the cluster;;;
---
09/Dec/22 00:32;jonvex;FileSystemBackedTableMetadata has config
{code:java}
DEFAULT_LISTING_PARALLELISM = 1500; {code};;;
---
09/Dec/22 00:45;jonvex;TimelineServerPerf has numExecuters with a default of 10
But then also has numCoresPerExecutor also with a default of 10
Something seems off here. Maybe it's supposed to be numExecutors per core? Whatever it is, those configs seem to conflict;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with a global search for HoodieEngineContext#flatMap, HoodieEngineContext#map, and similar APIs that accept parallelism, then inspect each caller and its parallelism configuration. Review FileSystemBackedTableMetadata's DEFAULT_LISTING_PARALLELISM and TimelineServerPerf's numExecuters and numCoresPerExecutor settings. Done means occurrences consistently use user-configured, cluster-appropriate parallelism rather than item counts, with conflicting defaults resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100