@gen_cluster with many nannies causes flakiness
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Any test decorated as follows:
```python
@gen_cluster(client=True, nthreads=[("", 1)] * 10, Worker=Nanny)
```
is extremely flaky on CI, regardless of environment; the message is
```
distributed.utils_test - ERROR - Failed to start gen_cluster: TimeoutError: Nanny failed to start in 15 seconds
```
This issue is glaring with 10 nannies, but I suspect it may also be happening more sporadically with less of them.
See for example https://github.com/dask/distributed/runs/4805666018?check_suite_focus=true :
```
E tornado.util.TimeoutError: Operation timed out after 60 seconds
../../../miniconda3/envs/dask-distributed/lib/python3.9/site-packages/tornado/ioloop.py:529: TimeoutError
----------------------------- Captured stderr call -----------------------------
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.utils_test - ERROR - Failed to start gen_cluster: TimeoutError: Nanny failed to start in 15 seconds; retrying
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/nanny.py", line 338, in start
response = await self.instantiate()
File "/Users/runner/work/distributed/distributed/distributed/nanny.py", line 407, in instantiate
result = await asyncio.wait_for(
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 468, in wait_for
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/core.py", line 264, in _
await asyncio.wait_for(self.start(), timeout=timeout)
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/utils_test.py", line 951, in coro
s, ws = await start_cluster(
File "/Users/runner/work/distributed/distributed/distributed/utils_test.py", line 852, in start_cluster
await asyncio.gather(*workers)
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 692, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/Users/runner/work/distributed/distributed/distributed/core.py", line 268, in _
raise TimeoutError(
asyncio.exceptions.TimeoutError: Nanny failed to start in 15 seconds
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.utils_test - ERROR - Failed to start gen_cluster: TimeoutError: Nanny failed to start in 15 seconds; retrying
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/nanny.py", line 338, in start
response = await self.instantiate()
File "/Users/runner/work/distributed/distributed/distributed/nanny.py", line 407, in instantiate
result = await asyncio.wait_for(
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 468, in wait_for
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/core.py", line 264, in _
await asyncio.wait_for(self.start(), timeout=timeout)
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/utils_test.py", line 951, in coro
s, ws = await start_cluster(
File "/Users/runner/work/distributed/distributed/distributed/utils_test.py", line 852, in start_cluster
await asyncio.gather(*workers)
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 692, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/Users/runner/work/distributed/distributed/distributed/core.py", line 268, in _
raise TimeoutError(
asyncio.exceptions.TimeoutError: Nanny failed to start in 15 seconds
distributed.worker - INFO - Stopping worker
distributed.worker - INFO - Closed worker has not yet started: Status.undefined
distributed.worker - INFO - Stopping worker
distributed.worker - INFO - Closed worker has not yet started: Status.undefined
distributed.worker - INFO - Stopping worker
distributed.worker - INFO - Closed worker has not yet started: Status.undefined
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 0 seconds, killing
distributed.utils_test - ERROR - Failed to start gen_cluster: TimeoutError: Nanny failed to start in 15 seconds; retrying
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/nanny.py", line 338, in start
response = await self.instantiate()
File "/Users/runner/work/distributed/distributed/distributed/nanny.py", line 407, in instantiate
result = await asyncio.wait_for(
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 468, in wait_for
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/core.py", line 264, in _
await asyncio.wait_for(self.start(), timeout=timeout)
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/runner/work/distributed/distributed/distributed/utils_test.py", line 951, in coro
s, ws = await start_cluster(
File "/Users/runner/work/distributed/distributed/distributed/utils_test.py", line 852, in start_cluster
await asyncio.gather(*workers)
File "/Users/runner/miniconda3/envs/dask-distributed/lib/python3.9/asyncio/tasks.py", line 692, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/Users/runner/work/distributed/distributed/distributed/core.py", line 268, in _
raise TimeoutError(
asyncio.exceptions.TimeoutError: Nanny failed to start in 15 seconds
```
Contributor guide
Assessment
This issue has not been assessed yet.