register workers of scheduler are less than workers in dashborad
- Dominant language
- Python
- Stars
- 69
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
**What happened**:
- we used the below way to start dask cluster with 20 workers
```python
n_workers = 20
spec_worker = skein.Service(resources=skein.Resources(memory=memory_limit, vcores=threads_per_worker),
instances=n_workers,
files=resource_files,
script=worker_script)
services["dask.worker"] = spec_worker
spec_scheduler = skein.Service(resources=skein.Resources(memory="20g", vcores=4),
files=resource_files,
instances=1,
script=scheduler_script)
services["dask.scheduler"] = spec_scheduler
spec = skein.ApplicationSpec(name=app_name,
queue=_get_queue(),
master=spec_master,
services=services)
cluster = YarnCluster.from_specification(spec)
```
- scheduler log showed only few worker registered

- but we can see all workers in dashborad

**What you expected to happen**:
all of 20 workers registered into scheduler
**Anything else we need to know?**:
- sometime scheduler showed all of 20 workers registered, but sometime not.
- submitted task always ran into issue as below when scheduler showed only part of 20 workers registered

**Environment**:
- Dask version: 2.19.0
- Python version: 3.6.7
- Operating System: CentOS Linux release 7.2.1511 (Core)
- Install method (conda, pip, source): conda
Contributor guide
Assessment
This issue has not been assessed yet.