fsspec / fsspec/filesystem_spec
Custom Caching Filesystem does not work with a url string
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I am creating a custom redis caching filesystem and I found that opening a file with a custom string does not work because of this hardcoded sequence https://github.com/fsspec/filesystem_spec/blob/0bb3f26c412d7ad9b2d52a5c32265014709d1c1f/fsspec/core.py#L352 :
bit = cls._strip_protocol(bit)
if (
protocol in {"blockcache", "filecache", "simplecache"}
and "target_protocol" not in kw
):
bit = previous_bit
out.append((bit, protocol, kw))
previous_bit = bit
Is there a way to work through this without forking fsspec? I am have registered my class and am trying to open like this:
import fsspec
from redis_fsspec_cache.sync import RedisCachingFileSystem
fsspec.register_implementation("rediscache", RedisCachingFileSystem)
with fsspec.open(
"rediscache::s3://nextgen-dmac-cloud-ingest/nos/ngofs2/nos.ngofs2.fields.best.nc.zarr",
mode="r",
s3={"anon": True},
rediscache={"redis_port": 6380},
) as f:
print(len(f.read()))
Opening through the filesystem works fine. Thank you for any help you can give!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec/core.py at the linked protocol-parsing sequence and reproduce the provided rediscache::s3://... call using the registration and keyword arguments shown. Compare it with direct filesystem access; done means the custom caching filesystem can open the URL string successfully without forking fsspec.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100