alibaba / alibaba/GraphScope

[BUG] [Graphscope on yarn] When generating Fragments, there is a partition issue.

Open
#3,772 0 comments 0 reactions 1 assignee Claimed by @zhanglei1949 View on GitHub
Dominant language
C++
Stars
3.6k
Forks
468
Avg merge
29m
Merged PRs (30d)
1

Description

**Describe the bug**
1.I have 3 machines.The initialization of executors in Spark is evenly distributed across different machines.
![image](https://github.com/alibaba/GraphScope/assets/11552872/077642e2-2b8c-4b6b-8b3b-d584278412ba)

2.When starting to generate the metadata for the FragmentRDD,the metadata information is skewed to one particular machine
![image](https://github.com/alibaba/GraphScope/assets/11552872/b1a084be-e752-4820-ac9f-8bdd6472edb0)

3.After analysis, the problematic code is likely to be here。There is an issue with the initialization of variables `rddhosts`、`rddLocations` and `partitionIds`. The partition acquisition mechanism here needs to be checked.
![image](https://github.com/alibaba/GraphScope/assets/11552872/3dc77054-2629-411d-8180-6a8d50e10625)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.