`CommClosedErrors` resulting from `Client.shutdown()`
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**Describe the issue**:
I am experiencing `CommClosedError` errors during the shutdown of a manually constructed Dask cluster using the Dask CLI. This was first noted in dask/dask-mpi#94.
**Minimal Complete Verifiable Example**:
You need a minimum of 3 open terminal sessions (e.g., bash) to reproduce the error.
In **Terminal 1**:
```bash
$ dask-scheduler
```
Note the `ADDRESS` in the `Scheduler at: ADDRESS:8786` log message.
In **Terminal 2**:
```bash
$ dask-worker ADDRESS:8786
```
where `ADDRESS` is the scheduler IP address (without the port number).
In **Terminal 3**:
```bash
$ python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from distributed import Client
>>> client = Client('ADDRESS:8786')
>>> client.shutdown()
```
where `ADDRESS` is the scheduler IP address noted above.
#### Results
In **Terminal 1**, the scheduler shuts down appropriately without errors.
In **Terminal 2**, the worker shuts down, but not without error. The logs of the worker after `client.shutdown()` is called are:
```text
2022-10-25 15:54:14,502 - distributed.worker - INFO - Stopping worker at ADDRESS:40141
2022-10-25 15:54:14,503 - distributed.worker - INFO - Connection to scheduler broken. Closing without reporting. ID: Worker-ce265b58-eecf-4335-bd1e-d0fc3b07e93d Address ADDRESS:40141 Status: Status.closing
2022-10-25 15:54:14,503 - distributed.batched - INFO - Batched Comm Closed Scheduler local=ADDRESS:33682 remote=ADDRESS:8786>
Traceback (most recent call last):
File ".../lib/python3.10/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File ".../lib/python3.10/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File ".../lib/python3.10/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
2022-10-25 15:54:14,507 - distributed.nanny - INFO - Worker closed
2022-10-25 15:54:14,508 - distributed.nanny - ERROR - Worker process died unexpectedly
2022-10-25 15:54:14,699 - distributed.nanny - INFO - Closing Nanny at 'ADDRESS:46611'.
2022-10-25 15:54:14,699 - distributed.dask_worker - INFO - End worker
```
In **Terminal 3**, where the `client` is running, a `CommClosedError` appears every time the client heartbeat is called (about once every 5 seconds):
```text
2022-10-25 15:54:15,825 - tornado.application - ERROR - Exception in callback >
Traceback (most recent call last):
File ".../lib/python3.10/site-packages/tornado/ioloop.py", line 905, in _run
return self.callback()
File ".../lib/python3.10/site-packages/distributed/client.py", line 1390, in _heartbeat
self.scheduler_comm.send({"op": "heartbeat-client"})
File ".../lib/python3.10/site-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm Scheduler local=ADDRESS:33802 remote=ADDRESS:8786> already closed.
```
And this does not stop repeating until the Python process is exited (e.g., `exit()`).
**Anything else we need to know?**:
Interestingly, you can avoid the error that appears in **Terminal 2** in the worker logs if you call `client.retire_workers()` before calling `client.shutdown()`, but the `CommClosedError` errors in the Client application (**Terminal 3**) are still present.
**NOTE: The `CommClosedError` messages in the worker logs appears to have been introduced in version 2022.4.2. These errors do not appear in version 2022.4.1 or 2022.4.0.**
**Environment**:
- Dask version: 2022.10.0 --> 2022.4.2 / Client heartbeat `CommClosedErrors` appear much further back
- Python version: 3.10.X, 3.9.X, 3.8.X
- Operating System: Linux, Windows
- Install method (conda, pip, source): conda (from conda-forge)
Contributor guide
Assessment
This issue has not been assessed yet.