dask / dask/distributed

Broken Pipe when shutting down a simple Client() on KeyboardInterrupt

Open
#3,384 14 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Python 3.6.2
dask==2.9.1
distributed==2.9.1

I'm trying to shutdown a client when the main thread catches a keyboard exception with the following code:

```
if __name__ == "__main__":

try:

client = Client()

# keep it running
while True:
pass

except KeyboardInterrupt as ke:

if client is not None:

print('Shutting down client')
client.shutdown()
```

However, I receive several connection errors when I do this:

```
^CShutting down client
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - ERROR - Nanny failed to start process
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
await self.process.start()
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
process.start()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/forkserver.py", line 66, in connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 61] Connection refused
distributed.nanny - ERROR - Nanny failed to start process
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
await self.process.start()
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
process.start()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/forkserver.py", line 66, in connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 61] Connection refused
distributed.nanny - ERROR - Nanny failed to start process
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
await self.process.start()
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
process.start()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/forkserver.py", line 66, in connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 61] Connection refused
distributed.nanny - ERROR - Nanny failed to start process
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
await self.process.start()
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
process.start()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/forkserver.py", line 66, in connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 61] Connection refused
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 247, in _feed
send_bytes(obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 247, in _feed
send_bytes(obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 247, in _feed
send_bytes(obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Task exception was never retrieved
future: exception=CommClosedError('in : ConnectionRefusedError: [Errno 61] Connection refused',)>
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/comm/tcp.py", line 361, in connect
ip, port, max_buffer_size=MAX_BUFFER_SIZE, **kwargs
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/tornado/tcpclient.py", line 280, in connect
af, addr, stream = await connector.start(connect_timeout=timeout)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/tornado/tcpclient.py", line 143, in on_connect_done
stream = future.result()
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/comm/tcp.py", line 373, in connect
convert_stream_closed_error(self, e)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/comm/tcp.py", line 130, in convert_stream_closed_error
raise CommClosedError("in %s: %s: %s" % (obj, exc.__class__.__name__, exc))
distributed.comm.core.CommClosedError: in : ConnectionRefusedError: [Errno 61] Connection refused
```

I've tried using client.shutdown() as well as client.stop() but nothing seems to be fixing it. I was also thinking that the scheduler might be getting closed before the workers, so I attempted to shutdown the workers first with

```
workers = list(client.scheduler_info()['workers'])
client.run_on_scheduler(lambda scheduler=None: scheduler.retire_workers(workers, close_workers=True))

client.shutdown()
```

But that just gives me `'NoneType' object has no attribute 'retire_workers'` as well as the stacktrace mentioned above. Any help would be greatly appreciated

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.