mars-project / mars-project/mars

What's best practice for setup cluster?

Open
#3,180 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

ENV: python 3.7.11 mars 0.9.0

```Python
import mars
import mars.tensor as mt

n = 20000
n_worker = 1
n_cpu = 1
mem_bytes = 20 * 2 ** 30

mars.new_session(init_local=True, n_worker=n_worker, n_cpu=n_cpu, mem_bytes=mem_bytes)

X = mt.random.RandomState(0).rand(n,n)
invX = mt.linalg.inv(X).execute()

```

While executing above script, the speed is positive correlate to `n_cpu` and negative correlate to `n_worker`.
Is it right result or I did something wrong?
If it was right, is that meaning I should always choose 1 worker with multi cpu , instead of multi worker with 1 cpu?
And what should I do if i can run on multi machines, but only a little cpu core every machine.

samples: execute time in seconds.

| Worker\CPU | 1 | 2 | 4 | 8 | 16 |
| :---: | :---: | :---: | :---: | :---: | :---: |
| 1 | 1100 | 546 | 291 | 184 | 124 |
| 2 | \ | 857 | 360 | 211 | 150 |
| 4 | \ | \ | 836 | 383 | 234 |
| 8 | \ | \ | \ | 417 | 310 |
| 16 | \ | \ | \ | \ | 555 |

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided Python benchmark using mars.new_session, varying n_worker and n_cpu across the reported matrix. Compare the observed execution times with the cluster setup guidance, and consider the multi-machine, low-CPU case; done means the worker/CPU relationship and recommended configuration are clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, performance
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.