drivendataorg / drivendataorg/cloudpathlib
Some libraries generate confusing errors when opening CloudPath objects
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
The main problem is the libs based on opening files with `fsspec` won't work properly with `CloudPaths`.
They will call `fspath` to resolve it (instead of str()) and therefore the file called will be cached (this is not exaclty what we want, and this will be done silently)
For single files like `tif` this will be not optimal but will work (your example)
But for nested files likes [VRT](https://gdal.org/drivers/raster/vrt.html), this will crash as only the header is cached and not the relative files. This will break the paths inside the VRT.
Note that GDAL (and therefore rasterio) can read VRT stored on the cloud, and this is the expected behavior.
A little example:
```python
vrt = AnyPath("https://s3.unistra.fr/merge_32-31.vrt")
rasterio.open(str(vrt)) # will work
rasterio.open(vrt) # will fail for the wrong reason, saying that the files linked inside the VRT don't exist
```
_Originally posted by @remi-braun in https://github.com/drivendataorg/cloudpathlib/issues/96#issuecomment-1402068539_
Contributor guide
Assessment
This issue has not been assessed yet.