dask / dask/distributed

distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:52230 remote=tcp://127.0.0.1:42125>

Open
#2,573 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I am getting this problem when using the `distributed` scheduler to compute 16,000,000 dot products. It makes the dashboard to not respond for several minutes and the computations are not started immediately. When they do start, I do not see all the resources of my workstations being used. I wonder why that would be.

I also see errors like the following ones:

```
remote=tcp://127.0.0.1:42125>
distributed.comm.tcp - WARNING - Closing dangling stream in

tornado.application - ERROR - Exception in callback >
Traceback (most recent call last):
File "/home/muammar/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/home/muammar/.local/lib/python3.7/site-packages/bokeh/server/tornado.py", line 542, in _keep_alive
c.send_ping()
File "/home/muammar/.local/lib/python3.7/site-packages/bokeh/server/connection.py", line 80, in send_ping
self._socket.ping(codecs.encode(str(self._ping_count), "utf-8"))
File "/home/muammar/.local/lib/python3.7/site-packages/tornado/websocket.py", line 447, in ping
raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError

```
I suspect, but I might be wrong, that those socket errors are because the dashboard is waiting for the new 16000000 delayed computations to be populated in an array that I pass to the scheduler. Therefore, for a couple of minutes, it does not have any data to show. Would that be a cause for that?

When this happens I am unable to `Ctrl-c` the script running into the problem and my computer might even freeze (this happened just once). I posted [more information here](https://stackoverflow.com/questions/55251699/how-to-use-client-scatter-correctly-and-when-in-dask).

Any idea on what is going on and how to solve it?

**Edit 1**: My workstation has 64GB RAM, and it is consumed completely. I will go through the best practices of dask.delayed to do this in batches and see if I get rid of the warnings above.

**Edit 2**: After doing the computations in batches I can get them done, but the matrix I build does not fit in memory and that is fine.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.