mars-project / mars-project/mars

[BUG] request quota hang

Open
#2,873 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version
2. The version of Mars you use
3. Versions of crucial packages, such as numpy, scipy and pandas
4. Full stack of the error.
```
2022-03-25 17:47:09,592 WARNING debug.py:81 -- Calling ('request_batch_quota', 0, ({('wpTn1PPhshAbKzts6OjyOwEW', 'rggQ1mQWCmsuHZywpERqSoXE'): 1902358070},), {}) on b'numa-0_quota' at ray://mars_cluster_1648201040/45/0 timed out(timeout for 210.0061 seconds).
```
![image](https://user-images.githubusercontent.com/12445254/160097941-f2695ba5-32e5-492f-9756-63bde375f217.png)
Node resource:
![image](https://user-images.githubusercontent.com/12445254/160098027-8321417e-413e-46e5-9ff4-52f185472c3d.png)

Contributor guide

Open the contributing guide

Research direction

Start with the request_batch_quota timeout reported at debug.py:81 and trace the call associated with b'numa-0_quota'. Use the provided Mars cluster, stack trace, and node-resource details to investigate the hang. Done means the quota request no longer times out under the reported conditions and the behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.