dask / dask/dask-yarn

dask-yarn not requesting the exact number of workers i specified

Open
#59 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
69
Forks
40
PR merge metrics
No merged PRs in 30d

Description

I'm running the following, very simple example, with dask-yarn.

```
cluster = YarnCluster(environment='venv:///home/florisvannee/my_envsd',
worker_vcores=1,
worker_memory="256MiB")
cluster.scale(1)
client = Client(cluster)

def inc(x):
return x + 1

a = client.submit(inc, 20)
print(a)
print(a.result())
```

This never completes. If I change the call to `scale `with any number higher than 1, it does complete. Eg. with `cluster.scale(2)` I get the following output in the log:

```
19/04/19 14:07:05 INFO skein.ApplicationMaster: REQUESTED: dask.scheduler_0
19/04/19 14:07:06 INFO skein.ApplicationMaster: Starting container_1555675382310_0003_01_000002...
19/04/19 14:07:06 INFO skein.ApplicationMaster: RUNNING: dask.scheduler_0 on container_1555675382310_0003_01_000002
19/04/19 14:07:07 INFO skein.ApplicationMaster: Scaling service 'dask.worker' to 2 instances, a delta of 2.
19/04/19 14:07:07 INFO skein.ApplicationMaster: REQUESTED: dask.worker_0
19/04/19 14:07:07 INFO skein.ApplicationMaster: REQUESTED: dask.worker_1
19/04/19 14:07:12 INFO skein.ApplicationMaster: Starting container_1555675382310_0003_01_000003...
19/04/19 14:07:12 INFO skein.ApplicationMaster: RUNNING: dask.worker_1 on container_1555675382310_0003_01_000003
```

Even though it looks like it is requesting worker_0, it never gets it. When I look in the overall resourcemanager log, I never see any request for worker_0 though. I only see requests for worker 1. Note that this is not, because there is in this example no work (for more complex examples with more load I see exactly the same behavior, the first worker never gets properly requested). Also, when running with `cluster.scale(1)`, it never completes because that first worker is just never started up.
Running `scale()` with any number higher than 2 always results in all workers being started except worker_0.

Any ideas what could be causing this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.