fsspec / fsspec/filesystem_spec

differences between gcsfs and fsspec behavior when opening gs:// url paths

Open
#476 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I would like to open a gcs path using fsspec's url resolver and then read it with xarray:

import xarray as xr
import fsspec

url = 'gs://ldeo-glaciology/bedmachine/BedMachineAntarctica_2019-11-05_v01.nc'  
openfile = fsspec.open(url, mode='rb') 
dsgcs = xr.open_dataset(openfile, chunks=3000)

this raises the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-26-9f11dae45dd1> in <module>
      1 url = 'gs://ldeo-glaciology/bedmachine/BedMachineAntarctica_2019-11-05_v01.nc'
      2 openfile = fsspec.open(url, mode='rb')
----> 3 dsgcs = xr.open_dataset(openfile, chunks=3000)
      4 
      5 get_ipython().run_line_magic('time', 'dsgcs.surface.mean().compute()')

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
    534                 "with engine='scipy' or 'h5netcdf'"
    535             )
--> 536         engine = _get_engine_from_magic_number(filename_or_obj)
    537         if engine == "scipy":
    538             store = backends.ScipyDataStore(filename_or_obj, **backend_kwargs)

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/api.py in _get_engine_from_magic_number(filename_or_obj)
    113         magic_number = filename_or_obj[:8]
    114     else:
--> 115         if filename_or_obj.tell() != 0:
    116             raise ValueError(
    117                 "file-like object read/write pointer not at zero "

AttributeError: 'OpenFile' object has no attribute 'tell'

However, if I do the same thing with gcsfs, it works

import gcsfs
gcs = gcsfs.GCSFileSystem()

openfile = gcs.open(url, mode='rb') 
dsgcs = xr.open_dataset(openfile, chunks=3000)

This feels like a bug. And it breaks my mental model of how fsspec works. I thought that fsspec was just dispatching to gcsfs based on url matching. Help me understand why that is not the case.

xref https://github.com/pydata/xarray/issues/4591, which helped me discover this (but is about something different)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the fsspec.open(url, mode='rb') entry point and compare its returned OpenFile with the object returned by gcsfs.GCSFileSystem.open. Check how xarray.open_dataset consumes each object, then add or update a focused regression test if the project contains one for this behavior. Done means the differing interfaces are explained and the reported gs:// workflow has a clear, tested outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.