`Client.close()` noisy
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
@ncclementi reported offline that calling `Client.close()` leads to a lot of printed output. Here's a simple reproducer
```python
In [1]: from distributed import Client
In [2]: c = Client()
In [3]: c.close()
2023-06-07 14:30:41,763 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/worker.py", line 1238, in heartbeat
response = await retry_operation(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 434, in retry_operation
return await retry(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 413, in retry
return await coro()
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1365, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1124, in send_recv
response = await comm.read(deserializers=deserializers)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in : Stream is closed
2023-06-07 14:30:41,866 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/worker.py", line 1238, in heartbeat
response = await retry_operation(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 434, in retry_operation
return await retry(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 413, in retry
return await coro()
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1365, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1124, in send_recv
response = await comm.read(deserializers=deserializers)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in : Stream is closed
2023-06-07 14:30:41,866 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 225, in read
frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/james/projects/dask/distributed/distributed/worker.py", line 1238, in heartbeat
response = await retry_operation(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 434, in retry_operation
return await retry(
File "/Users/james/projects/dask/distributed/distributed/utils_comm.py", line 413, in retry
return await coro()
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1365, in send_recv_from_rpc
return await send_recv(comm=comm, op=key, **kwargs)
File "/Users/james/projects/dask/distributed/distributed/core.py", line 1124, in send_recv
response = await comm.read(deserializers=deserializers)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 241, in read
convert_stream_closed_error(self, e)
File "/Users/james/projects/dask/distributed/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in : Stream is closed
```
Going through our git history manually it looks like https://github.com/dask/distributed/pull/7730 is where this started happening. It's not immediately clear how the changes in that PR are related, but locally if I uninstall `gilknocker` or set `dask.config.set({"distributed.admin.system-monitor.gil.enabled": False})` then I consistently don't get the noisy output.
cc @milesgranger @graingert @hendrikmakait
Contributor guide
Assessment
This issue has not been assessed yet.