dask / dask/distributed

ValueError: invalid operation on non-started TCPListener when ending script

Open
#4,477 3 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

When I distribute the work using SLURMCluster to launch some workers, I get this error when the program exits.

``` python
tornado.application - ERROR - Exception in callback functools.partial(. at 0x7f284335f5e0>, exception=ValueError('invalid operation on non-started TCPListener')>)
Traceback (most recent call last):
File "/software/opt/bionic/x86_64/python/3.8-2020.11/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/software/opt/bionic/x86_64/python/3.8-2020.11/lib/python3.8/site-packages/tornado/tcpserver.py", line 327, in
gen.convert_yielded(future), lambda f: f.result()
File ".../venv/lib/python3.8/site-packages/distributed/comm/tcp.py", line 465, in _handle_stream
logger.debug("Incoming connection from %r to %r", address, self.contact_address)
File ".../venv/lib/python3.8/site-packages/distributed/comm/tcp.py", line 501, in contact_address
host, port = self.get_host_port()
File ".../venv/lib/python3.8/site-packages/distributed/comm/tcp.py", line 482, in get_host_port
self._check_started()
File ".../venv/lib/python3.8/site-packages/distributed/comm/tcp.py", line 457, in _check_started
raise ValueError("invalid operation on non-started TCPListener")
ValueError: invalid operation on non-started TCPListener
```

It looks like all the computation its actually done, but I don't manage to understand why I get this error.

As suggested in [this issue](https://stackoverflow.com/questions/53394935/what-is-the-right-way-to-close-a-dask-localcluster) I tried to use a context manager to see if this would help, but I obtain the same issue.

The actual work consists on delayed tasks that read and write files using xarray and the error does not happen if not using SLURMCluster to initiate the workers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.