Broken Pipe when shutting down a simple Client() on KeyboardInterrupt
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Python 3.6.2
dask==2.9.1
distributed==2.9.1
I'm trying to shutdown a client when the main thread catches a keyboard exception with the following code:
```
if __name__ == "__main__":
try:
client = Client()
# keep it running
while True:
pass
except KeyboardInterrupt as ke:
if client is not None:
print('Shutting down client')
client.shutdown()
```
However, I receive several connection errors when I do this:
```
^CShutting down client
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - ERROR - Nanny failed to start process
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
await self.process.start()
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
process.start()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/forkserver.py", line 66, in connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 61] Connection refused
distributed.nanny - ERROR - Nanny failed to start process
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
await self.process.start()
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
process.start()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/forkserver.py", line 66, in connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 61] Connection refused
distributed.nanny - ERROR - Nanny failed to start process
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
await self.process.start()
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
process.start()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/forkserver.py", line 66, in connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 61] Connection refused
distributed.nanny - ERROR - Nanny failed to start process
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/nanny.py", line 522, in start
await self.process.start()
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/process.py", line 202, in _start
process.start()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_forkserver.py", line 51, in _launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/forkserver.py", line 66, in connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 61] Connection refused
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 247, in _feed
send_bytes(obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 247, in _feed
send_bytes(obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 247, in _feed
send_bytes(obj)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Task exception was never retrieved
future: exception=CommClosedError('in : ConnectionRefusedError: [Errno 61] Connection refused',)>
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/comm/tcp.py", line 361, in connect
ip, port, max_buffer_size=MAX_BUFFER_SIZE, **kwargs
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/tornado/tcpclient.py", line 280, in connect
af, addr, stream = await connector.start(connect_timeout=timeout)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/tornado/tcpclient.py", line 143, in on_connect_done
stream = future.result()
tornado.iostream.StreamClosedError: Stream is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/comm/tcp.py", line 373, in connect
convert_stream_closed_error(self, e)
File "/Users/jordanmiles/Documents/CTPDataLake/env/lib/python3.6/site-packages/distributed/comm/tcp.py", line 130, in convert_stream_closed_error
raise CommClosedError("in %s: %s: %s" % (obj, exc.__class__.__name__, exc))
distributed.comm.core.CommClosedError: in : ConnectionRefusedError: [Errno 61] Connection refused
```
I've tried using client.shutdown() as well as client.stop() but nothing seems to be fixing it. I was also thinking that the scheduler might be getting closed before the workers, so I attempted to shutdown the workers first with
```
workers = list(client.scheduler_info()['workers'])
client.run_on_scheduler(lambda scheduler=None: scheduler.retire_workers(workers, close_workers=True))
client.shutdown()
```
But that just gives me `'NoneType' object has no attribute 'retire_workers'` as well as the stacktrace mentioned above. Any help would be greatly appreciated
Contributor guide
Assessment
This issue has not been assessed yet.