mars-project / mars-project/mars
What's best practice for setup cluster?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
ENV: python 3.7.11 mars 0.9.0
```Python
import mars
import mars.tensor as mt
n = 20000
n_worker = 1
n_cpu = 1
mem_bytes = 20 * 2 ** 30
mars.new_session(init_local=True, n_worker=n_worker, n_cpu=n_cpu, mem_bytes=mem_bytes)
X = mt.random.RandomState(0).rand(n,n)
invX = mt.linalg.inv(X).execute()
```
While executing above script, the speed is positive correlate to `n_cpu` and negative correlate to `n_worker`.
Is it right result or I did something wrong?
If it was right, is that meaning I should always choose 1 worker with multi cpu , instead of multi worker with 1 cpu?
And what should I do if i can run on multi machines, but only a little cpu core every machine.
samples: execute time in seconds.
| Worker\CPU | 1 | 2 | 4 | 8 | 16 |
| :---: | :---: | :---: | :---: | :---: | :---: |
| 1 | 1100 | 546 | 291 | 184 | 124 |
| 2 | \ | 857 | 360 | 211 | 150 |
| 4 | \ | \ | 836 | 383 | 234 |
| 8 | \ | \ | \ | 417 | 310 |
| 16 | \ | \ | \ | \ | 555 |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided Python benchmark using mars.new_session, varying n_worker and n_cpu across the reported matrix. Compare the observed execution times with the cluster setup guidance, and consider the multi-machine, low-CPU case; done means the worker/CPU relationship and recommended configuration are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, performance
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100