fsspec / fsspec/filesystem_spec
Add `get_protocols`, `strip_protocols`, and `is_fully_qualified` APIs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I have a use case for the following APIs, and wanted to get opinions on whether they are feasible as additions:
Firstly,
def get_protocols(url: str) -> tuple[str, ...]:
"""Returns all protocols found on a URL."""
-> get_protocols("my-bucket/a.txt") = ()
-> get_protocols("s3://my-bucket/a.txt") = ("s3",)
-> get_protocols("filecache::s3://my-bucket/a.txt") = ("filecache", "s3")
Then,
def strip_protocols(url: str) -> str:
"""Strips all protocols found on a URL."""
-> strip_protocols("s3://my-bucket/a.txt") = "my-bucket/a.txt"
-> strip_protocols("filecache::s3://my-bucket/a.txt") = "my-bucket/a.txt"
And finally,
def is_fully_qualified(url: str) -> bool:
"""Whether a given url has a leading protocol."""
-> is_fully_qualified("my-bucket/a.txt") = False
-> is_fully_qualified("s3://my-bucket/a.txt") = True
-> is_fully_qualified("filecache::s3://my-bucket/a.txt") = True
I believe existing APIs (especially regarding protocol stripping) only strip the first found protocol. The other stuff is more related to the fact that the existing get_protocol returns "file" if no protocol is found, which is not ideal if you want to check whether the given URL is prefixed with an actual scheme.
My use case is that I want to initialize the correct target file system based on a input URI, and supporting cache prefixes already requires number 1) because I also need the target protocol.
Furthermore, I pad input paths with a configurable user prefix, and before initializing the file system, I need to assert that the resulting padded path is a full URI, otherwise the filesystem lookup fails.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing get_protocol API and the current protocol-stripping behavior. Compare the proposed get_protocols, strip_protocols, and is_fully_qualified examples with existing tests, then clarify the intended semantics and add coverage for unprefixed URLs, single protocols, and nested cache prefixes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100