mars-project / mars-project/mars

[BUG] mars shuffle function not well-distributed

Open
#2,934 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Groupby shuffle keys for different groups are not well-distributed. In a online case which has 10000_0000 lines and chunk size is 20_0000, some gorups has about 24000 keys, but most groups has less than 5000 keys. The overall pecess is dominated by large keys group, and the execution is 5 times slower than expected.
![image](https://user-images.githubusercontent.com/12445254/163951195-049c7226-7368-4ff6-9829-21887b9a9e85.png)
![image](https://user-images.githubusercontent.com/12445254/163951998-2ef1e470-1feb-44dd-9445-7da206ea0fc0.png)

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: 3.7
2. The version of Mars you use: master
3. Versions of crucial packages, such as numpy, scipy and pandas
4. Full stack of the error.
5. Minimized code to reproduce the error.

**Expected behavior**
The keys should be well-distributed. This is not a data skew. For data skew, some key groups will have much more data thant other group, but the issue is that some chunks has much more keys than other chunks.

Contributor guide

Open the contributing guide

Research direction

Start by locating Mars's groupby shuffle implementation and the code that assigns shuffle keys to chunks. Reproduce the reported imbalance with Python 3.7, master, 100000000 lines, and chunk size 200000, then compare key counts across chunks. Done means group keys are distributed more evenly without treating the case as ordinary data skew.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.