apache / apache/hudi

Use proper parallelism for engine context APIs

Open
#15,587 1 comment 0 reactions 0 assignees View on GitHub
area:performance from-jira priority:critical status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

do a global search of these APIs
- org.apache.hudi.common.engine.HoodieEngineContext#flatMap
- org.apache.hudi.common.engine.HoodieEngineContext#map

and similar ones take in parallelism.

A lot of occurrences are using number of items as parallelism, which affect performance. Parallelism should be based on num cores available in the cluster and set by user via parallelism configs.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5261
- Type: Improvement

---

## Comments

08/Dec/22 23:47;jonvex;I see in this guide [https://spark.apache.org/docs/3.2.1/rdd-programming-guide.html#resilient-distributed-datasets-rdds] that it suggests 2-4 per core in the cluster;;;

---

09/Dec/22 00:32;jonvex;FileSystemBackedTableMetadata has config 
{code:java}
DEFAULT_LISTING_PARALLELISM = 1500; {code};;;

---

09/Dec/22 00:45;jonvex;TimelineServerPerf has numExecuters with a default of 10

But then also has numCoresPerExecutor also with a default of 10

Something seems off here. Maybe it's supposed to be numExecutors per core? Whatever it is, those configs seem to conflict;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with a global search for HoodieEngineContext#flatMap, HoodieEngineContext#map, and similar APIs that accept parallelism, then inspect each caller and its parallelism configuration. Review FileSystemBackedTableMetadata's DEFAULT_LISTING_PARALLELISM and TimelineServerPerf's numExecuters and numCoresPerExecutor settings. Done means occurrences consistently use user-configured, cluster-appropriate parallelism rather than item counts, with conflicting defaults resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.