dask / dask/distributed

`Client.close()` noisy

Open
#7,891 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

@ncclementi reported offline that calling `Client.close()` leads to a lot of printed output. Here's a simple reproducer

```python
In [1]: from distributed import Client

In [2]: c = Client()

In [3]: c.close()
2023-06-07 14:30:41,763 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/worker.py", line 1238, in heartbeat
response = await retry_operation(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 434, in retry_operation
return await retry(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 413, in retry
return await coro()
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1365, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1124, in send_recv
response = await comm.read(deserializers=deserializers)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in : Stream is closed
2023-06-07 14:30:41,866 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/worker.py", line 1238, in heartbeat
response = await retry_operation(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 434, in retry_operation
return await retry(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 413, in retry
return await coro()
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1365, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1124, in send_recv
response = await comm.read(deserializers=deserializers)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in : Stream is closed
2023-06-07 14:30:41,866 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/worker.py", line 1238, in heartbeat
response = await retry_operation(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 434, in retry_operation
return await retry(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 413, in retry
return await coro()
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1365, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1124, in send_recv
response = await comm.read(deserializers=deserializers)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in : Stream is closed
```

Going through our git history manually it looks like https://github.com/dask/distributed/pull/7730 is where this started happening. It's not immediately clear how the changes in that PR are related, but locally if I uninstall `gilknocker` or set `dask.config.set({"distributed.admin.system-monitor.gil.enabled": False})` then I consistently don't get the noisy output.

cc @milesgranger @graingert @hendrikmakait

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.