fsspec / fsspec/filesystem_spec
differences between gcsfs and fsspec behavior when opening gs:// url paths
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I would like to open a gcs path using fsspec's url resolver and then read it with xarray:
import xarray as xr
import fsspec
url = 'gs://ldeo-glaciology/bedmachine/BedMachineAntarctica_2019-11-05_v01.nc'
openfile = fsspec.open(url, mode='rb')
dsgcs = xr.open_dataset(openfile, chunks=3000)
this raises the following error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-26-9f11dae45dd1> in <module>
1 url = 'gs://ldeo-glaciology/bedmachine/BedMachineAntarctica_2019-11-05_v01.nc'
2 openfile = fsspec.open(url, mode='rb')
----> 3 dsgcs = xr.open_dataset(openfile, chunks=3000)
4
5 get_ipython().run_line_magic('time', 'dsgcs.surface.mean().compute()')
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
534 "with engine='scipy' or 'h5netcdf'"
535 )
--> 536 engine = _get_engine_from_magic_number(filename_or_obj)
537 if engine == "scipy":
538 store = backends.ScipyDataStore(filename_or_obj, **backend_kwargs)
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/api.py in _get_engine_from_magic_number(filename_or_obj)
113 magic_number = filename_or_obj[:8]
114 else:
--> 115 if filename_or_obj.tell() != 0:
116 raise ValueError(
117 "file-like object read/write pointer not at zero "
AttributeError: 'OpenFile' object has no attribute 'tell'
However, if I do the same thing with gcsfs, it works
import gcsfs
gcs = gcsfs.GCSFileSystem()
openfile = gcs.open(url, mode='rb')
dsgcs = xr.open_dataset(openfile, chunks=3000)
This feels like a bug. And it breaks my mental model of how fsspec works. I thought that fsspec was just dispatching to gcsfs based on url matching. Help me understand why that is not the case.
xref https://github.com/pydata/xarray/issues/4591, which helped me discover this (but is about something different)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the fsspec.open(url, mode='rb') entry point and compare its returned OpenFile with the object returned by gcsfs.GCSFileSystem.open. Check how xarray.open_dataset consumes each object, then add or update a focused regression test if the project contains one for this behavior. Done means the differing interfaces are explained and the reported gs:// workflow has a clear, tested outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100