bytedance / bytedance/CloudShuffleService

分区太多会导致driver内存溢出

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
260
Forks
57
PR merge metrics
No merged PRs in 30d

Description

spark 分区数据量特别大20多万的分区,driver内存为15G,由于master是通过driver启动的,作业运行1/3之后,shuffleEpochSetMap 占用内存将近10G左右,导致driver内存溢出,shuffleEpochSetMap 的大小跟什么有关系,为什么会占用这么多的内存,有什么解决办法吗

Contributor guide

Open the contributing guide

Research direction

Start by tracing how shuffleEpochSetMap is created and retained during a Spark job with more than 200,000 partitions, focusing on the driver and master interaction described in the issue. Measure its growth against job progress and partition count. Done means explaining the memory relationship and identifying a supported way to prevent the driver from exhausting its 15G allocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.