fsspec / fsspec/filesystem_spec

Better support urllike objects in utils.stringify_path

Open
#696 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

When passing a pydantic AnyUrl (https://pydantic-docs.helpmanual.io/usage/types/#urls) to some of the s3fs methods it seems to lop off the bucket portion of the path.

Whilst this can easily be alleviated by making use of str(url) prior to interacting to s3fs/fsspec I wonder if we can have better support for url-like objects passed to the fsspec methods

Presently in https://github.com/intake/filesystem_spec/blob/9c75d6267d930b4fb01333d22bc2bebf4836116c/fsspec/utils.py#L312 we extract the path attribute if an object has one.

Should we have something a bit higher level that can deal with objects that have both a path and host/netloc attributes and treat those accordingly?

This would allow fsspec to neatly support pydantic url types as well as the types from urllib.parse

def stringify_path(filepath):
    if isinstance(filepath, str):
        return filepath
    elif hasattr(filepath, "__fspath__"):
        return filepath.__fspath__()
    elif isinstance(filepath, pathlib.Path):
        return str(filepath)
    elif hasattr(filepath, "path"):
        if hasattr(filepath, "host"):
            return  filepath.host + filepath.path
       elif hasattr(filepath, "netloc"):
            return  filepath.netloc + filepath.path            
        return filepath.path
    else:
        return filepath

Eg

In [1]: path = 's3://bucket/key'

In [2]: import pydantic, urllib.parse

In [3]: urllib.parse.urlparse('s3://bucket/key')
Out[3]: ParseResult(scheme='s3', netloc='bucket', path='/key', params='', query='', fragment='')

In [4]:  pydantic.parse_obj_as(pydantic.AnyUrl, path)
Out[4]: AnyUrl('s3://bucket/key', scheme='s3', host='bucket', host_type='int_domain', path='/key')

Additionally these objects do already contain the scheme if we want to use it

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/utils.py at stringify_path, then reproduce the reported behavior with a pydantic AnyUrl and urllib.parse.urlparse result for s3://bucket/key. Done means these URL-like objects retain the scheme and bucket/netloc or host instead of losing the bucket portion when passed to fsspec methods.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.