dask / dask/distributed

Cancelling task waiting on event breaks tornado

Open
#3,908 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
Tornado breaks.
**What you expected to happen**:
The task to cancel silently.
**Minimal Complete Verifiable Example**:
Run on jupyter lab 2.1.4, on a cell block.
PS: does not produce error if run on a worker (see comment below).

```python
from dask.distributed import Client
client = await Client(asynchronous=True)

from distributed import Event
import asyncio
e=await Event()
t = asyncio.create_task(e.wait())
await asyncio.sleep(1)
t.cancel()
```

**Anything else we need to know?**:
The traceback (repeats every few seconds):

```
tornado.application - ERROR - Exception in callback functools.partial(>, exception=AssertionError('Already reading')>)
```

```
tornado.application - ERROR - Exception in callback functools.partial(>, exception=AssertionError('Already reading')>)
Traceback (most recent call last):
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
future.result()
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/client.py", line 1141, in _update_scheduler_info
self._scheduler_identity = await self.scheduler.identity()
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/core.py", line 806, in send_recv_from_rpc
result = await send_recv(comm=comm, op=key, **kwargs)
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/core.py", line 589, in send_recv
response = await comm.read(deserializers=deserializers)
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 184, in read
n_frames = await stream.read_bytes(8)
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/iostream.py", line 436, in read_bytes
future = self._start_read()
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/iostream.py", line 824, in _start_read
assert self._read_future is None, "Already reading"
AssertionError: Already reading
tornado.application - ERROR - Exception in callback functools.partial(>, exception=AssertionError('Already reading')>)
Traceback (most recent call last):
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
future.result()
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/client.py", line 1141, in _update_scheduler_info
self._scheduler_identity = await self.scheduler.identity()
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/core.py", line 806, in send_recv_from_rpc
result = await send_recv(comm=comm, op=key, **kwargs)
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/core.py", line 589, in send_recv
response = await comm.read(deserializers=deserializers)
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 184, in read
n_frames = await stream.read_bytes(8)
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/iostream.py", line 436, in read_bytes
future = self._start_read()
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/iostream.py", line 824, in _start_read
assert self._read_future is None, "Already reading"
AssertionError: Already reading
tornado.application - ERROR - Exception in callback functools.partial(>, exception=AssertionError('Already reading')>)
Traceback (most recent call last):
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
future.result()
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/client.py", line 1141, in _update_scheduler_info
self._scheduler_identity = await self.scheduler.identity()
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/core.py", line 806, in send_recv_from_rpc
result = await send_recv(comm=comm, op=key, **kwargs)
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/core.py", line 589, in send_recv
response = await comm.read(deserializers=deserializers)
File "/home/ma/miniconda3/lib/python3.7/site-packages/distributed/comm/tcp.py", line 184, in read
n_frames = await stream.read_bytes(8)
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/iostream.py", line 436, in read_bytes
future = self._start_read()
File "/home/ma/miniconda3/lib/python3.7/site-packages/tornado/iostream.py", line 824, in _start_read
assert self._read_future is None, "Already reading"
```

**Environment**:

- Dask version: 2.18.1
- Python version: 3.7
- Operating System: Linux, WSL
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.