dask-yarn not requesting the exact number of workers i specified
- Dominant language
- Python
- Stars
- 69
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
I'm running the following, very simple example, with dask-yarn.
```
cluster = YarnCluster(environment='venv:///home/florisvannee/my_envsd',
worker_vcores=1,
worker_memory="256MiB")
cluster.scale(1)
client = Client(cluster)
def inc(x):
return x + 1
a = client.submit(inc, 20)
print(a)
print(a.result())
```
This never completes. If I change the call to `scale `with any number higher than 1, it does complete. Eg. with `cluster.scale(2)` I get the following output in the log:
```
19/04/19 14:07:05 INFO skein.ApplicationMaster: REQUESTED: dask.scheduler_0
19/04/19 14:07:06 INFO skein.ApplicationMaster: Starting container_1555675382310_0003_01_000002...
19/04/19 14:07:06 INFO skein.ApplicationMaster: RUNNING: dask.scheduler_0 on container_1555675382310_0003_01_000002
19/04/19 14:07:07 INFO skein.ApplicationMaster: Scaling service 'dask.worker' to 2 instances, a delta of 2.
19/04/19 14:07:07 INFO skein.ApplicationMaster: REQUESTED: dask.worker_0
19/04/19 14:07:07 INFO skein.ApplicationMaster: REQUESTED: dask.worker_1
19/04/19 14:07:12 INFO skein.ApplicationMaster: Starting container_1555675382310_0003_01_000003...
19/04/19 14:07:12 INFO skein.ApplicationMaster: RUNNING: dask.worker_1 on container_1555675382310_0003_01_000003
```
Even though it looks like it is requesting worker_0, it never gets it. When I look in the overall resourcemanager log, I never see any request for worker_0 though. I only see requests for worker 1. Note that this is not, because there is in this example no work (for more complex examples with more load I see exactly the same behavior, the first worker never gets properly requested). Also, when running with `cluster.scale(1)`, it never completes because that first worker is just never started up.
Running `scale()` with any number higher than 2 always results in all workers being started except worker_0.
Any ideas what could be causing this?
Contributor guide
Assessment
This issue has not been assessed yet.