apache / apache/arrow

[Python] unexpected URL encoded path (white spaces) when uploading to S3

Open
#34,905 11 comments 1 reaction 0 assignees View on GitHub
Component: Python good-first-issue Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.
#### Environment

OS: Windows/Linux
Python: 3.10.10
s3fs: from 2022.7.1 to 2023.3.0 (doesn't matter)
S3 Backend: MinIO / Ceph (doesn't matter)

#### Description

Version 11.0.0 of pyarrow introduced an unexpected behavior when uploading Parquet Files to an S3 Bucket (using s3fs.S3FileSystem), if the Path to the Parque File contains white spaces. White Spaces will be replaced by URL encoded Syntax %20 e.g:
A Directory Name like:

> product=My Fancy Product

becomes

> product=My%20Fancy%20Product

on S3 filesystem. **NOTICE**: the Equal Sign `=` is URL encoded for the request, but won't become %3D on S3 filesystem. That means, the URL encoded equal sign `=` seems to be interpreted correctly

#### Example Code
```python
# s3fs FileSystem Object
def return_s3filesystem(url, user, pw):
fs = s3fs.S3FileSystem(
anon=False,
use_ssl=True,
client_kwargs={
"endpoint_url": url,
"aws_access_key_id": user,
"aws_secret_access_key": pw,
"verify": False,
}
)
return fs

def write_df_to_s3(df, partition_cols, path_to_s3_object, url, user, pw, more_than_one_date_per_file,
delete_parquet_files):
'''
write Parquet File from Pandas DataFrame to S3 Bucket
'''

# instantiate s3fs.S3FileSystem object
fs = return_s3filesystem(url, user, pw)
# if the parquet file allready exists, delete it if requested, to prevent duplicated data
delete_if_exists(fs, path_to_s3_object, df, more_than_one_date_per_file, delete_existing_files=delete_parquet_files)
try:
# create ArrowTable from DataFrame
arrow_table = Table.from_pandas(df)
except ArrowTypeError as e:
# this is Error No. 1626701451158
raise InvalidDataFrame(errorno=1626701451158, dataframe=df, arrowexception=e)
except TypeError as e:
raise InvalidDataFrame(errorno=1627657641211, dataframe=df, arrowexception=e)
try:
# write Parquet File to S3 Bucket, using S3FileSystem object 'fs' from above. Create directories by partition_cols
pq.write_to_dataset(arrow_table,
path_to_s3_object,
partition_cols=partition_cols,
filesystem=fs,
use_dictionary=False,
data_page_size=100000,
compression="snappy",
version="2.0")
except ArrowTypeError as e:
raise InvalidDataFrame(errorno=1627575189, dataframe=df, arrowexception=e)
except aiohttp.client_exceptions.ClientConnectionError as e:
raise S3ConnectionError(errorno=1627575130, exmsg=e)

```

#### Example Result
##### Expected Result (using pyarrow 10.0.1)
![image](https://user-images.githubusercontent.com/122540156/230012242-0076a82a-f985-495f-a2a4-851928f42a3d.png)

###### Debug output
```
botocore.endpoint - DEBUG - Sending http request:
botocore.parsers - DEBUG - Response headers: HTTPHeaderDict({'accept-ranges': 'bytes', 'cache-control': 'no-cache', 'content-length': '471', 'content-security-policy': 'block-all-mixed-content', 'content-t
ype': 'application/xml', 'etag': '"caca775951f07ca64f530aae539fe5cd-3"', 'server': 'MinIO', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'vary': 'Accept-Encoding', 'x-accel-buffering
': 'no', 'x-amz-id-2': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'x-amz-request-id': '1752F9747004F3E5', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block',
'date': 'Wed, 05 Apr 2023 07:31:29 GMT'})
botocore.parsers - DEBUG - Response body:
b'\nhttp://localhost:9000/my-products/product=My%20Fancy%20Product/date=2023-0
1-05/0d5d1f2c5032472dbad1d17c845d5432-0.parquetmy-productsproduct=My Fancy Product/date=2023-01-05/0d5d1f2c5032472dbad1d17c845d5432-0.parquet"caca775951f07c
a64f530aae539fe5cd-3"'
botocore.hooks - DEBUG - Event needs-retry.s3.CompleteMultipartUpload: calling handler
botocore.hooks - DEBUG - Event needs-retry.s3.CompleteMultipartUpload: calling handler
botocore.retryhandler - DEBUG - No retry needed.
botocore.hooks - DEBUG - Event needs-retry.s3.CompleteMultipartUpload: calling handler >

```

##### Actual result (using pyarrow 11.0.0)
![image](https://user-images.githubusercontent.com/122540156/230014637-03f361dd-7a9b-4309-9a8e-9ad106c4ca23.png)

###### Debug output
```
botocore.endpoint - DEBUG - Sending http request:
botocore.parsers - DEBUG - Response headers: HTTPHeaderDict({'accept-ranges': 'bytes', 'cache-control': 'no-cache', 'content-length': '479', 'content-security-policy': 'block-all-mixed-content', 'content-t
ype': 'application/xml', 'etag': '"f44ab58edcc877c4d00075b9db28e4e5-3"', 'server': 'MinIO', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'vary': 'Accept-Encoding', 'x-accel-buffering
': 'no', 'x-amz-id-2': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'x-amz-request-id': '1752F9DC0E0CE8AD', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block',
'date': 'Wed, 05 Apr 2023 07:38:54 GMT'})
botocore.parsers - DEBUG - Response body:
b'\nhttp://localhost:9000/my-products/product=My%2520Fancy%2520Product/date=20
23-01-10/a724b93c251a486b897eb7b151c622bd-0.parquetmy-productsproduct=My%20Fancy%20Product/date=2023-01-10/a724b93c251a486b897eb7b151c622bd-0.parquet"f44ab5
8edcc877c4d00075b9db28e4e5-3"'
botocore.hooks - DEBUG - Event needs-retry.s3.CompleteMultipartUpload: calling handler
botocore.hooks - DEBUG - Event needs-retry.s3.CompleteMultipartUpload: calling handler
botocore.retryhandler - DEBUG - No retry needed.
botocore.hooks - DEBUG - Event needs-retry.s3.CompleteMultipartUpload: calling handler >
```
The difference in the debug output is the line starting with **botocore.parsers - DEBUG - Response body:**. In the XML Part, the Node `` contains an URL Encoded string (pyarrow 11.0.0) vs. "human readable" String (pyarrow 10.0.1). But the URL encoded string is not URL encoded at all, as mentioned before e.g. the equal sign `=` is intepreted as expected.

It seems, that the URL encode/decode(?) isn't done correctly at all?

Wild guess of mine: This behavior might be introduced by: #33598 and/or #33468

Thanks,
Sven

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the return_s3filesystem and write_df_to_s3 examples, then compare the pyarrow 10.0.1 and 11.0.0 debug output for paths containing spaces. Reproduce the multipart upload against the S3-compatible MinIO or Ceph backend; done means the stored S3 key preserves the readable path while the request remains correctly encoded.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.