fsspec / fsspec/filesystem_spec

Use of '::' in file's name

Open
#1,782 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I am investigating this issue in uproot, where we try to open a file that contains :: in the name of the file. As you can see, the expected behavior would be to correctly create the file indicated in the string, but it doesn't seem to be the case. The reason is in this function
https://github.com/fsspec/filesystem_spec/blob/fe59f48363029a0da68a0de534bf63e1c42e5a81/fsspec/core.py#L331-L366
where, it seems to me, the case of :: being part of the file name is not considered, and it is only treated as a protocol separator.

One idea to adapt the code would be the following:

    if "::" in path:
        x = re.compile(".*[^a-z]+.*")  # test for non protocol-like single word
        bits = []
        for p in path.split("::"):
            # Check if part looks like a protocol or URL
            if "://" in p or x.match(p) or p in known_implementations:
                bits.append(p)
            else:
                # If not, assume it is part of the file name
                bits.append(p + "://")
        
        # If no part matches a known protocol, treat the entire path as a file name
        if not any(b for b in bits if b.strip("://") in known_implementations):
            bits = [path]
    else:
        bits = [path]

This fixes Jim's reproducer, but breaks a few tests, making me wonder if this behavior is intentional.

The question is thus the following: should a logic be implemented in fsspec to handle the case in which :: is part of the file name or should we implement a check in uproot where we raise an error if :: is not used as a protocol separator?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fsspec/core.py lines 331-366 and the linked uproot issue 1251, then reproduce the filename containing ::. Run the affected tests, including the cases that the proposed parsing change breaks. Done requires a decided behavior for :: and regression coverage for both filenames and protocol separators.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.