drivendataorg / drivendataorg/cloudpathlib
`OverwriteNewerLocalError` when reading same resource in parallel
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
This appears to be similar to #128, with two caveats:
1) This happens despite using the workaround mentioned by @jayqi in that issue
2) There is no writing happening, only reading
The relevant code is:
```
# self.image_path is a cloud path
with self.image_path.open("rb") as file:
image_contents = typing.cast(np.ndarray, imageio.imread(file))
```
This line raises the following exception:
```
cloudpathlib.exceptions.OverwriteNewerLocalError: Local file (...) for cloud path (...) is newer on disk, but is being requested for download from cloud. Either (1) push your changes to the cloud, (2) remove the local file, or (3) pass `force_overwrite_from_cloud=True` to overwrite.
```
This is despite the fact the program does not write to the file or even open it in 'w' mode.
What may be relevant is that the code is part of an HTTP server which receives several requests in parallel, and it often needs to read the same files for different requests.
Since reading the image file may be a lengthy process which probably uses C code extensively, I think it's possible `open` gets called from another thread before the `with` clause is released, which causes the issue.
Is this a known issue? Does the explanation make sense, or am I missing something else?
Contributor guide
Assessment
This issue has not been assessed yet.