dask / dask/distributed

`CommClosedErrors` resulting from `Client.shutdown()`

Open
#7,192 14 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:

I am experiencing `CommClosedError` errors during the shutdown of a manually constructed Dask cluster using the Dask CLI. This was first noted in dask/dask-mpi#94.

**Minimal Complete Verifiable Example**:

You need a minimum of 3 open terminal sessions (e.g., bash) to reproduce the error.

In **Terminal 1**:
```bash
$ dask-scheduler
```

Note the `ADDRESS` in the `Scheduler at: ADDRESS:8786` log message.

In **Terminal 2**:
```bash
$ dask-worker ADDRESS:8786
```

where `ADDRESS` is the scheduler IP address (without the port number).

In **Terminal 3**:
```bash
$ python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from distributed import Client
>>> client = Client('ADDRESS:8786')
>>> client.shutdown()
```

where `ADDRESS` is the scheduler IP address noted above.

#### Results

In **Terminal 1**, the scheduler shuts down appropriately without errors.

In **Terminal 2**, the worker shuts down, but not without error. The logs of the worker after `client.shutdown()` is called are:

```text
2022-10-25 15:54:14,502 - distributed.worker - INFO - Stopping worker at ADDRESS:40141
2022-10-25 15:54:14,503 - distributed.worker - INFO - Connection to scheduler broken. Closing without reporting. ID: Worker-ce265b58-eecf-4335-bd1e-d0fc3b07e93d Address ADDRESS:40141 Status: Status.closing
2022-10-25 15:54:14,503 - distributed.batched - INFO - Batched Comm Closed Scheduler local=ADDRESS:33682 remote=ADDRESS:8786>
Traceback (most recent call last):
File ".../lib/python3.10/site-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File ".../lib/python3.10/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File ".../lib/python3.10/site-packages/distributed/comm/tcp.py", line 269, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
2022-10-25 15:54:14,507 - distributed.nanny - INFO - Worker closed
2022-10-25 15:54:14,508 - distributed.nanny - ERROR - Worker process died unexpectedly
2022-10-25 15:54:14,699 - distributed.nanny - INFO - Closing Nanny at 'ADDRESS:46611'.
2022-10-25 15:54:14,699 - distributed.dask_worker - INFO - End worker
```

In **Terminal 3**, where the `client` is running, a `CommClosedError` appears every time the client heartbeat is called (about once every 5 seconds):

```text
2022-10-25 15:54:15,825 - tornado.application - ERROR - Exception in callback >
Traceback (most recent call last):
File ".../lib/python3.10/site-packages/tornado/ioloop.py", line 905, in _run
return self.callback()
File ".../lib/python3.10/site-packages/distributed/client.py", line 1390, in _heartbeat
self.scheduler_comm.send({"op": "heartbeat-client"})
File ".../lib/python3.10/site-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm Scheduler local=ADDRESS:33802 remote=ADDRESS:8786> already closed.
```

And this does not stop repeating until the Python process is exited (e.g., `exit()`).

**Anything else we need to know?**:

Interestingly, you can avoid the error that appears in **Terminal 2** in the worker logs if you call `client.retire_workers()` before calling `client.shutdown()`, but the `CommClosedError` errors in the Client application (**Terminal 3**) are still present.

**NOTE: The `CommClosedError` messages in the worker logs appears to have been introduced in version 2022.4.2. These errors do not appear in version 2022.4.1 or 2022.4.0.**

**Environment**:

- Dask version: 2022.10.0 --> 2022.4.2 / Client heartbeat `CommClosedErrors` appear much further back
- Python version: 3.10.X, 3.9.X, 3.8.X
- Operating System: Linux, Windows
- Install method (conda, pip, source): conda (from conda-forge)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.