drivendataorg / drivendataorg/cloudpathlib

`OverwriteNewerLocalError` when reading same resource in parallel

Open
#283 8 comments 1 reaction 0 assignees View on GitHub
caching
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

This appears to be similar to #128, with two caveats:
1) This happens despite using the workaround mentioned by @jayqi in that issue
2) There is no writing happening, only reading

The relevant code is:
```
# self.image_path is a cloud path
with self.image_path.open("rb") as file:
image_contents = typing.cast(np.ndarray, imageio.imread(file))
```

This line raises the following exception:
```
cloudpathlib.exceptions.OverwriteNewerLocalError: Local file (...) for cloud path (...) is newer on disk, but is being requested for download from cloud. Either (1) push your changes to the cloud, (2) remove the local file, or (3) pass `force_overwrite_from_cloud=True` to overwrite.
```

This is despite the fact the program does not write to the file or even open it in 'w' mode.

What may be relevant is that the code is part of an HTTP server which receives several requests in parallel, and it often needs to read the same files for different requests.
Since reading the image file may be a lengthy process which probably uses C code extensively, I think it's possible `open` gets called from another thread before the `with` clause is released, which causes the issue.

Is this a known issue? Does the explanation make sense, or am I missing something else?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.