dask / dask/dask-yarn

Cluster and Client creation error when using EMR

Open
#124 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
69
Forks
40
PR merge metrics
No merged PRs in 30d

Description

**What happened**:

Running dask-yarn on EMR causes a repeating error in tornado on client creation.

**What you expected to happen**:

No error, just the client being created and being able to use it.

**Minimal Complete Verifiable Example**:

I am using dask-yarn on EMR. I followed the directions outlined [here](https://yarn.dask.org/en/latest/aws-emr.html) and used the unaltered [bootstrap-dask](https://github.com/dask/dask-yarn/blob/master/deployment_resources/aws-emr/bootstrap-dask) script from this repo. I [used the `emr-5.29.0` release](https://github.com/dask/dask-yarn/issues/122#issuecomment-669362550) to avoid the other bootstrap issue #122. Connect to the master node and open jupyter notebook. Start with a new notebook.

```python
from dask_yarn import YarnCluster
from dask.distributed import Client

# Create a cluster
cluster = YarnCluster()

# Connect to the cluster
client = Client(cluster)
client
```
I then get this error every few seconds reported back to the notebook:

```
distributed.scheduler - INFO - Receive client connection: Client-d994493a-d748-11ea-afbf-025317f23bc7
distributed.core - INFO - Starting established connection
Exception in callback with_timeout..error_callback() at /home/hadoop/miniconda/lib/python3.8/site-packages/tornado/gen.py:968
handle: .error_callback() at /home/hadoop/miniconda/lib/python3.8/site-packages/tornado/gen.py:968>
Traceback (most recent call last):
File "/home/hadoop/miniconda/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/home/hadoop/miniconda/lib/python3.8/site-packages/tornado/gen.py", line 970, in error_callback
future.result()
asyncio.exceptions.CancelledError
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2.22.0
- Python version: 3.8.3
- Operating System: emr-5.29.0 (Amazon linux)
- Install method (conda, pip, source): [conda from bootstrap-dask script](https://github.com/dask/dask-yarn/blob/aac7410666dacce5a97b4b8c621df9eb922519a8/deployment_resources/aws-emr/bootstrap-dask#L105-L114).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.