fsspec / fsspec/filesystem_spec
GCSFilesystem creation is surprisingly slow
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
this snippet takes ~24s to run on my laptop in Germany:
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "fsspec",
# "gcsfs",
# ]
# ///
from fsspec import url_to_fs
from time import time
url = 'gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3'
if __name__ == '__main__':
start = time()
_ = url_to_fs(url)
elapsed = time() - start
print(elapsed)
26s seems a bit long for string parsing -- clearly something else is going on inside this function -- but I'm a bit surprised that there's anything else going on side here, given the apparent simplicity of the URL. Is this expected behavior, or something that should be fixed?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Python script and tracing the fsspec.url_to_fs entry point for the gs:// URL and GCSFilesystem creation. Determine what accounts for the delay and whether that behavior is expected; done means documenting the cause or identifying a concrete, testable fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100