fsspec / fsspec/filesystem_spec
Better support urllike objects in utils.stringify_path
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
When passing a pydantic AnyUrl (https://pydantic-docs.helpmanual.io/usage/types/#urls) to some of the s3fs methods it seems to lop off the bucket portion of the path.
Whilst this can easily be alleviated by making use of str(url) prior to interacting to s3fs/fsspec I wonder if we can have better support for url-like objects passed to the fsspec methods
Presently in https://github.com/intake/filesystem_spec/blob/9c75d6267d930b4fb01333d22bc2bebf4836116c/fsspec/utils.py#L312 we extract the path attribute if an object has one.
Should we have something a bit higher level that can deal with objects that have both a path and host/netloc attributes and treat those accordingly?
This would allow fsspec to neatly support pydantic url types as well as the types from urllib.parse
def stringify_path(filepath):
if isinstance(filepath, str):
return filepath
elif hasattr(filepath, "__fspath__"):
return filepath.__fspath__()
elif isinstance(filepath, pathlib.Path):
return str(filepath)
elif hasattr(filepath, "path"):
if hasattr(filepath, "host"):
return filepath.host + filepath.path
elif hasattr(filepath, "netloc"):
return filepath.netloc + filepath.path
return filepath.path
else:
return filepath
Eg
In [1]: path = 's3://bucket/key'
In [2]: import pydantic, urllib.parse
In [3]: urllib.parse.urlparse('s3://bucket/key')
Out[3]: ParseResult(scheme='s3', netloc='bucket', path='/key', params='', query='', fragment='')
In [4]: pydantic.parse_obj_as(pydantic.AnyUrl, path)
Out[4]: AnyUrl('s3://bucket/key', scheme='s3', host='bucket', host_type='int_domain', path='/key')
Additionally these objects do already contain the scheme if we want to use it
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec/utils.py at stringify_path, then reproduce the reported behavior with a pydantic AnyUrl and urllib.parse.urlparse result for s3://bucket/key. Done means these URL-like objects retain the scheme and bucket/netloc or host instead of losing the bucket portion when passed to fsspec methods.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100