dask / dask/distributed

scatter without workers fails

Open
#2,454 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

This is with dask-jobqueue as the cluster for a Client.

given something similar to this:
```
myclust = SLURMCluster(cores=1, memory="2GB", queue='myqu')
myclust.adapt(minimum=0, maximum=10)
client = await Client(myclust, asychronous=True)
[myfut] = await client.scatter([mydata])
tasks = [client.submit(mytask, x, myfut) for x in range(20)]
```

i get the error
```
distributed.core - ERROR - No workers found
Traceback (most recent call last):
File "/home/danpf/git/distributed/distributed/core.py", line 346, in handle_comm
result = yield result
File "/home/danpf/.local/share/pyenv/versions/3.7.0/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/home/danpf/.local/share/pyenv/versions/3.7.0/lib/python3.7/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/danpf/git/distributed/distributed/scheduler.py", line 2277, in scatter
raise gen.TimeoutError("No workers found")
tornado.util.TimeoutError: No workers found
```

if you do this without being asynchronous it just stalls forever (or a long time i never waited that long)

A fix would be for scattering to impose a sort of 'number_of_cores' minimum, but looking at how ncores and so on are generated it doesn't appear to be an easy fix at fist glance.

briefly: adaptive clusters don't consider scatter jobs when determining the number of cores/workers they should have.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.