fsspec / fsspec/gcsfs

Getting 'GS' key error when reading a csv from GCS using gcsfc

Open
#162 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

Hi I upgraded gcsfs and now I get the following error:

My code is pretty simple:

data = dd.read_csv(file_path, parse_dates=[date_column])\
        .compute()
    return data```

It used to work but all of a sudden it stopped working.
file_path = gs://mybuck/res.csv

```File "main.py", line 51, in run
    data = load_parse_file(file_path=args.input_file)
  File "/FbProphet/prophet_gcp/utils.py", line 15, in load_parse_file
    data = dd.read_csv(file_path, parse_dates=[date_column])\
  File "/work/miniconda/lib/python3.7/site-packages/dask/dataframe/io/csv.py", line 578, in read
    **kwargs
  File "/work/miniconda/lib/python3.7/site-packages/dask/dataframe/io/csv.py", line 405, in read_pandas
    **(storage_options or {})
  File "/work/miniconda/lib/python3.7/site-packages/dask/bytes/core.py", line 93, in read_bytes
    fs, fs_token, paths = get_fs_token_paths(urlpath, mode="rb", storage_options=kwargs)
  File "/work/miniconda/lib/python3.7/site-packages/dask/bytes/core.py", line 425, in get_fs_token_paths
    fs, fs_token = get_fs(protocol, options)
  File "/work/miniconda/lib/python3.7/site-packages/dask/bytes/core.py", line 571, in get_fs
    cls = _filesystems[protocol]
KeyError: 'gs'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported dask.dataframe.read_csv call and the traceback through dask/bytes/core.py, especially get_fs_token_paths and get_fs. Reproduce the gs:// read with the upgraded gcsfs setup and compare filesystem registration behavior; done means the CSV loads without KeyError: 'gs'.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.