dask / dask/distributed

"Stream is closed"

Open
#1,674 37 comments 25 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I am having workers die with the following error messages.

This occurs when I am trying to `.persist()` large xarray dataset into memory. (There is more than enough memory in the cluster for the dataset by a factor of 5.)

I don't know what these errors mean, other than that the workers have died. Advice would be appreciated on how to debug more effectively.

```
distributed.worker - ERROR - failed during get data
Traceback (most recent call last):
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/comm/tcp.py", line 221, in write
yield future
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/worker.py", line 524, in get_data
compressed = yield comm.write(msg)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/comm/tcp.py", line 225, in write
convert_stream_closed_error(self, e)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/comm/tcp.py", line 124, in convert_stream_closed_error
raise CommClosedError("in %s: %s: %s" % (obj, exc.__class__.__name__, exc))
distributed.comm.core.CommClosedError: in : ConnectionResetError: [Errno 104] Connection reset by peer
distributed.core - WARNING - Lost connection to 'tcp://10.43.8.25:48251': in : ConnectionResetError: [Errno 104] Connection reset by peer
distributed.worker - ERROR - Worker stream died during communication: tcp://10.43.4.25:53234
Traceback (most recent call last):
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/comm/tcp.py", line 182, in read
frame = yield stream.read_bytes(length)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/worker.py", line 1763, in gather_dep
who=self.address)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/core.py", line 516, in send_recv_from_rpc
result = yield send_recv(comm=comm, op=key, **kwargs)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/core.py", line 350, in send_recv
response = yield comm.read()
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/comm/tcp.py", line 188, in read
convert_stream_closed_error(self, e)
File "/rigel/ocp/users/ra2697/conda/envs/pangeo/lib/python3.6/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in : Stream is closed
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.