fsspec / fsspec/filesystem_spec

lpath of filesystem.get doesn't handle pathlib.Path gracefully

Open Beginner friendly
#896 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I didn't find an existing issue, so I am opening a new one.

It looks like we can't pass pathlib Path's as local paths for get. I've only tried it against the GitHub implementation so far but considering that it fails in what appears to be a general utility function it might affect other implementations, too. Here is a reproducing example of the problem:

>>> import fsspec
>>> from pathlib import Path
>>> foo = Path("test_file.md") 
>>> fs = fsspec.filesystem("github", org="fsspec", repo="filesystem_spec")
>>> fs.get("README.md", foo)      
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Sebastian\AppData\Local\Programs\Python\Python38\lib\site-packages\fsspec\spec.py", line 771, in get
    lpaths = other_paths(rpaths, lpath)
  File "C:\Users\Sebastian\AppData\Local\Programs\Python\Python38\lib\site-packages\fsspec\utils.py", line 380, in other_paths
    assert len(paths) == len(path2)
TypeError: object of type 'WindowsPath' has no len()
>>> fs.get("README.md", str(foo))  # this works fine
>>>

Would it be possible to allow inputs of type pathlib.path for lpath? (e.g., by simply converting them to posix path strings: lpath.as_posix())

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with get in fsspec/spec.py and the other_paths utility in fsspec/utils.py, using the pathlib.Path reproduction from the issue. Verify that a Path can be used as lpath while the existing string input continues to work; completion is demonstrated when the reproduction no longer raises the WindowsPath len() TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.