fsspec / fsspec/filesystem_spec
Opening remote datasets not lazy with fsspec
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I'm finding that the following code reads all of the remote datasets into memory:
fs = gcsfs.GCSFileSystem(project='ldeo-glaciology', mode='ab')
filelist = fs.ls('gs://ldeo-glaciology/AMPS/wrf_d03_20161222_week-cf')
arrays = []
for filename in filelist:
with fsspec.open('gs://' + filename, mode='rb') as openfile:
arrays.append(xr.open_dataset(openfile, chunks = {}))
Is this the expected behavior? Is there a way to open these files as a remote dataset without fully reading them into memory? Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided fsspec.open and xr.open_dataset example with the listed GCS paths, then inspect how remote file handles are passed between them. Determine whether the observed memory loading is expected and, if not, identify the relevant behavior to change; done means establishing lazy access or documenting the limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100