apache / apache/arrow

[Python] ParquetWriter doesn't support all valid s3 paths

Open
#46,924 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

It appears `pyarrow.parquet.ParquetWriter` doesn't accept s3 path that contain `#` despite that being a valid character in s3 paths.

```python
>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>> schema = pa.schema([pa.field("foobar", pa.string())])
>>> writer = pq.ParquetWriter('s3://my-bucket/a#b#c.parquet', schema=schema)
Traceback (most recent call last):
File "", line 1, in
File "/project/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1014, in __init__
sink = self.file_handle = filesystem.open_output_stream(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_fs.pyx", line 885, in pyarrow._fs.FileSystem.open_output_stream
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected a local filesystem path, got a URI: 's3://my-bucket/a#b#c.parquet'
```

Other special characters (`[$*&@!]`)seem to be acceptable, so classifying this as a bug rather than a feature request.

via pyarrow v20.0.0

Possibly related to https://github.com/apache/arrow/issues/41365.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure through pyarrow.parquet.ParquetWriter with the shown s3:// path, then trace its call to filesystem.open_output_stream in pyarrow._fs. Done means a valid S3 path containing # is accepted without regressing the other special characters described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.