dask / dask/dask-yarn

register workers of scheduler are less than workers in dashborad

Open
#148 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
69
Forks
40
PR merge metrics
No merged PRs in 30d

Description

**What happened**:
- we used the below way to start dask cluster with 20 workers
```python
n_workers = 20
spec_worker = skein.Service(resources=skein.Resources(memory=memory_limit, vcores=threads_per_worker),
instances=n_workers,
files=resource_files,
script=worker_script)
services["dask.worker"] = spec_worker
spec_scheduler = skein.Service(resources=skein.Resources(memory="20g", vcores=4),
files=resource_files,
instances=1,
script=scheduler_script)
services["dask.scheduler"] = spec_scheduler

spec = skein.ApplicationSpec(name=app_name,
queue=_get_queue(),
master=spec_master,
services=services)

cluster = YarnCluster.from_specification(spec)
```
- scheduler log showed only few worker registered
![image](https://user-images.githubusercontent.com/22552116/124513494-ce87cc00-dd8f-11eb-842f-969e37519486.png)
- but we can see all workers in dashborad
![image](https://user-images.githubusercontent.com/22552116/124513541-df384200-dd8f-11eb-8b1e-3a9963ffa24c.png)

**What you expected to happen**:
all of 20 workers registered into scheduler

**Anything else we need to know?**:
- sometime scheduler showed all of 20 workers registered, but sometime not.
- submitted task always ran into issue as below when scheduler showed only part of 20 workers registered
![image](https://user-images.githubusercontent.com/22552116/124513832-85844780-dd90-11eb-9389-c008e8191e87.png)

**Environment**:
- Dask version: 2.19.0
- Python version: 3.6.7
- Operating System: CentOS Linux release 7.2.1511 (Core)
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.