Warning about scratch directories creating time taking too long
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
In one of my latest stress tests I encountered a warning (windows, obviously) about the time it takes to create a scratch directory
```
C:\Miniconda3\envs\dask-distributed\lib\contextlib.py:142: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
```
followed by a failing nanny start due to a timeout of 15s
```python
2022-06-27 21:01:13,838 - distributed.utils_test - ERROR - Failed to start gen_cluster: TimeoutError: Nanny start timed out after 15s.; retrying
Traceback (most recent call last):
File "d:\a\distributed\distributed\distributed\nanny.py", line 357, in start_unsafe
response = await self.instantiate()
File "d:\a\distributed\distributed\distributed\nanny.py", line 416, in instantiate
result = await asyncio.wait_for(
File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 432, in wait_for
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 456, in wait_for
return fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "d:\a\distributed\distributed\distributed\core.py", line 491, in start
await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 458, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "d:\a\distributed\distributed\distributed\utils_test.py", line 1081, in async_fn
s, ws = await start_cluster(
File "d:\a\distributed\distributed\distributed\utils_test.py", line 924, in start_cluster
await asyncio.gather(*workers)
File "C:\Miniconda3\envs\dask-distributed\lib\asyncio\tasks.py", line 650, in _wrap_awaitable
return (yield from awaitable.__await__())
File "d:\a\distributed\distributed\distributed\core.py", line 494, in start
raise asyncio.TimeoutError(
asyncio.exceptions.TimeoutError: Nanny start timed out after 15s.
```
https://github.com/fjetter/distributed/runs/7077483712?check_suite_focus=true
cc @graingert
Contributor guide
Assessment
This issue has not been assessed yet.