fsspec / fsspec/filesystem_spec

Add `get_protocols`, `strip_protocols`, and `is_fully_qualified` APIs

Open
#1,461 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I have a use case for the following APIs, and wanted to get opinions on whether they are feasible as additions:

Firstly,

def get_protocols(url: str) -> tuple[str, ...]:
    """Returns all protocols found on a URL."""

-> get_protocols("my-bucket/a.txt") = ()
-> get_protocols("s3://my-bucket/a.txt") = ("s3",)
-> get_protocols("filecache::s3://my-bucket/a.txt") = ("filecache", "s3")

Then,

def strip_protocols(url: str) -> str:
    """Strips all protocols found on a URL."""

-> strip_protocols("s3://my-bucket/a.txt") = "my-bucket/a.txt"
-> strip_protocols("filecache::s3://my-bucket/a.txt") = "my-bucket/a.txt"

And finally,

def is_fully_qualified(url: str) -> bool:
    """Whether a given url has a leading protocol."""

-> is_fully_qualified("my-bucket/a.txt") = False
-> is_fully_qualified("s3://my-bucket/a.txt") = True
-> is_fully_qualified("filecache::s3://my-bucket/a.txt") = True

I believe existing APIs (especially regarding protocol stripping) only strip the first found protocol. The other stuff is more related to the fact that the existing get_protocol returns "file" if no protocol is found, which is not ideal if you want to check whether the given URL is prefixed with an actual scheme.

My use case is that I want to initialize the correct target file system based on a input URI, and supporting cache prefixes already requires number 1) because I also need the target protocol.
Furthermore, I pad input paths with a configurable user prefix, and before initializing the file system, I need to assert that the resulting padded path is a full URI, otherwise the filesystem lookup fails.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing get_protocol API and the current protocol-stripping behavior. Compare the proposed get_protocols, strip_protocols, and is_fully_qualified examples with existing tests, then clarify the intended semantics and add coverage for unprefixed URLs, single protocols, and nested cache prefixes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.