fsspec / fsspec/filesystem_spec

Opening remote datasets not lazy with fsspec

Open
#374 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I'm finding that the following code reads all of the remote datasets into memory:

fs = gcsfs.GCSFileSystem(project='ldeo-glaciology', mode='ab')
filelist = fs.ls('gs://ldeo-glaciology/AMPS/wrf_d03_20161222_week-cf')

arrays = []
for filename in filelist:
    with  fsspec.open('gs://' + filename, mode='rb') as openfile:  
        arrays.append(xr.open_dataset(openfile, chunks = {}))

Is this the expected behavior? Is there a way to open these files as a remote dataset without fully reading them into memory? Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided fsspec.open and xr.open_dataset example with the listed GCS paths, then inspect how remote file handles are passed between them. Determine whether the observed memory loading is expected and, if not, identify the relevant behavior to change; done means establishing lazy access or documenting the limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.