bytedance / bytedance/CloudShuffleService
分区太多会导致driver内存溢出
Open
- Dominant language
- Java
- Stars
- 260
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
spark 分区数据量特别大20多万的分区,driver内存为15G,由于master是通过driver启动的,作业运行1/3之后,shuffleEpochSetMap 占用内存将近10G左右,导致driver内存溢出,shuffleEpochSetMap 的大小跟什么有关系,为什么会占用这么多的内存,有什么解决办法吗
Contributor guide
Research direction
Start by tracing how shuffleEpochSetMap is created and retained during a Spark job with more than 200,000 partitions, focusing on the driver and master interaction described in the issue. Measure its growth against job progress and partition count. Done means explaining the memory relationship and identifying a supported way to prevent the driver from exhausting its 15G allocation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100