drivendataorg / drivendataorg/cloudpathlib

Some libraries generate confusing errors when opening CloudPath objects

Open
#315 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

The main problem is the libs based on opening files with `fsspec` won't work properly with `CloudPaths`.
They will call `fspath` to resolve it (instead of str()) and therefore the file called will be cached (this is not exaclty what we want, and this will be done silently)

For single files like `tif` this will be not optimal but will work (your example)
But for nested files likes [VRT](https://gdal.org/drivers/raster/vrt.html), this will crash as only the header is cached and not the relative files. This will break the paths inside the VRT.
Note that GDAL (and therefore rasterio) can read VRT stored on the cloud, and this is the expected behavior.

A little example:
```python

vrt = AnyPath("https://s3.unistra.fr/merge_32-31.vrt")
rasterio.open(str(vrt)) # will work
rasterio.open(vrt) # will fail for the wrong reason, saying that the files linked inside the VRT don't exist
```

_Originally posted by @remi-braun in https://github.com/drivendataorg/cloudpathlib/issues/96#issuecomment-1402068539_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.