asyncio.exceptions.TimeoutError: Nanny failed to start in 60 seconds
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
All tasks completely successfully in an adaptive cluster, but two worker pods start and then stop throwing:
```
2022-05-03 09:08:30,442 - distributed.nanny - INFO - Start Nanny at: 'tcp://10.144.194.72:42035'
2022-05-03 09:08:31,129 - distributed.worker - INFO - Start worker at: tcp://10.144.194.72:35109
2022-05-03 09:08:31,130 - distributed.worker - INFO - Listening to: tcp://10.144.194.72:35109
2022-05-03 09:08:31,130 - distributed.worker - INFO - dashboard at: 10.144.194.72:36607
2022-05-03 09:08:31,130 - distributed.worker - INFO - Waiting to connect to: tcp://dask-root-ea5e4f84-d.sandbox:8786
2022-05-03 09:08:31,130 - distributed.worker - INFO - -------------------------------------------------
2022-05-03 09:08:31,130 - distributed.worker - INFO - Threads: 1
2022-05-03 09:08:31,130 - distributed.worker - INFO - Memory: 3.73 GiB
2022-05-03 09:08:31,130 - distributed.worker - INFO - Local Directory: /workdir/dask-worker-space/worker-4mgp6l36
2022-05-03 09:08:31,130 - distributed.worker - INFO - -------------------------------------------------
2022-05-03 09:08:31,354 - distributed.worker - INFO - Stopping worker at tcp://10.144.194.72:35109
2022-05-03 09:09:30,437 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.144.194.72:42035'. Report closure to scheduler: None
2022-05-03 09:09:30,437 - distributed.nanny - INFO - Nanny asking worker to close
2022-05-03 09:09:31,241 - distributed.nanny - WARNING - Worker process still alive after 0.7999973297119141 seconds, killing
2022-05-03 09:09:31,242 - distributed.dask_worker - INFO - End worker
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/distributed/nanny.py", line 354, in start
response = await self.instantiate()
File "/usr/local/lib/python3.9/site-packages/distributed/nanny.py", line 413, in instantiate
result = await asyncio.wait_for(
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 466, in wait_for
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 490, in wait_for
return fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/distributed/core.py", line 299, in _
await asyncio.wait_for(self.start(), timeout=timeout)
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/dask-worker", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/distributed/cli/dask_worker.py", line 477, in main
loop.run_sync(run)
File "/usr/local/lib/python3.9/site-packages/tornado/ioloop.py", line 530, in run_sync
return future_cell[0].result()
File "/usr/local/lib/python3.9/site-packages/distributed/cli/dask_worker.py", line 471, in run
await asyncio.gather(*nannies)
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 688, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/usr/local/lib/python3.9/site-packages/distributed/core.py", line 303, in _
raise TimeoutError(
asyncio.exceptions.TimeoutError: Nanny failed to start in 60 seconds
2022-05-03 09:09:31,245 - distributed.process - INFO - reaping stray process
Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
Traceback (most recent call last):
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name=''> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x555a1dfb0750)
Current thread 0x00007f7702e8a740 (most recent call first):
```
**What you expected to happen**:
No exception during shutdown.
**Environment**:
Dask on Kuberenetes
- Dask version: dask 2022.4.2, dask-kubernetes 2021.10.0
- Python version: 3.9.12
- Operating System: Debian GNU/Linux 11 (bullseye)
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.