dask / dask/distributed

`FileNotFoundError` raised when trying to get data.

Open
#7,182 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue:**
My dask cluster sometimes raises `FileNotFoundError` saying that it cannot find some data that should have been spilled to disk before.

More detailed error log:
```
2022-10-24 17:10:10,260 - distributed.core - ERROR - Exception while handling op get_data
Traceback (most recent call last):
File "/home/xxx/miniconda3/lib/python3.8/site-packages/distributed/core.py", line 770, in _handle_comm
result = await result
File "/home/xxx/miniconda3/lib/python3.8/site-packages/distributed/worker.py", line 1672, in get_data
data = {k: self.data[k] for k in keys if k in self.data}
File "/home/xxx/miniconda3/lib/python3.8/site-packages/distributed/worker.py", line 1672, in
data = {k: self.data[k] for k in keys if k in self.data}
File "/home/xxx/miniconda3/lib/python3.8/site-packages/zict/buffer.py", line 106, in __getitem__
return self.slow_to_fast(key)
File "/home/xxx/miniconda3/lib/python3.8/site-packages/zict/buffer.py", line 92, in slow_to_fast
value = self.slow[key]
File "/home/xxx/miniconda3/lib/python3.8/site-packages/zict/cache.py", line 55, in __getitem__
value = self.data[key]
File "/home/xxx/miniconda3/lib/python3.8/site-packages/zict/func.py", line 55, in __getitem__
return self.load(self.d[key]) # type: ignore
File "/home/xxx/miniconda3/lib/python3.8/site-packages/zict/file.py", line 82, in __getitem__
with open(fn, "rb") as fh:
FileNotFoundError: [Errno 2] No such file or directory: '/home/xxx/dask/temp/226861_12/15/dask-worker-space/worker-_sfuwtgm/storage/%28%27getitem-d7b9e5f9aea9bf17f584ff3a8de8397e%27%2C%201%2C%200%29'
```

Sorry but I haven't figured out a way to reproduce the problem stably. Do you have any insights on what might cause this problem? Or is there a way to work around this (like re-compute the task if the local file no longer exists)? Thanks a lot.

**Minimal Complete Verifiable Example:**
**Environment:**

Dask version: 2022.8.0
Python version: 3.8
Operating System: ubuntu
Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.