mars-project / mars-project/mars
[BUG] Wrong memory usage detection on yarn
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
when deploying on yarn, or just start mars cluster in command line, we can see high memory usage on mars-web ui even without any task.
Memory usage detection is based on the whole machine for now, but it should only calculate memory usage by itself on yarn. Otherwise the error `w:0:MemQuotaActor met hard memory limitation: request 0, available -23158964224, hard limit 121388728320` will occur by mistake.
**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: 3.7.9
2. The version of Mars you use: pymars[distributed]==0.6.0
3. Versions of crucial packages, such as numpy, scipy and protobuf: numpy==1.19.4, scipy==1.5.4, protobuf==3.14.0, pyarrow==2.0.0
**Expected behavior**
Calculate memory usage by mars rather than the whole machine.
Contributor guide
Research direction
The issue names no source files or tests. Start by tracing memory-usage detection and MemQuotaActor handling for YARN deployments, then verify that usage is calculated for Mars rather than the whole machine and that the reported false hard memory-limit error no longer occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100