apache / apache/arrow

[Python] Pyarrow fs incorrectly resolves S3 URIs with white space as a local path

Open
#41,365 12 comments 1 reaction 1 assignee Claimed by @ernestprovo23 View on GitHub
Component: Python good-first-issue Status: needs champion Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Pyarrow fs incorrectly resolves valid S3 URIs with a whitespace as a local path:
```python
from pyarrow.fs import _resolve_filesystem_and_path, FileSystem

uri = "s3://bucket/prefix with space/a=a"

resolved_filesystem, resolved_path = _resolve_filesystem_and_path(uri, None)

resolved_filesystem

```

This causes subsequent calls such as getting the file info to fail:
```python
path_info = resolved_filesystem.get_file_info(resolved_path)

pyarrow.lib.ArrowInvalid: Expected a local filesystem path, got a URI...
```

A quick look into the [method](https://github.com/apache/arrow/blob/main/python/pyarrow/fs.py#L165) indicates that a LocalFilesytem is chosen by default and returned if alternative filesystems are not detected which seems like a dubious strategy...

I assume this is [where](https://github.com/apache/arrow/blob/main/python/pyarrow/fs.py#L179) the S3 filesystem should be detected but a URI containing a whitespace seems to throw an exception although it's valid:
```python
filesystem, path = FileSystem.from_uri(uri)

Cannot parse URI: 's3://bucket/prefix with space/a=a/'
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.