apache / apache/arrow

[Python/C++] Make upload size per part configurable when uploading to S3

Open
#40,420 4 comments 3 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Currently, the size of every part in a multipart upload is hard-coded here: https://github.com/apache/arrow/blob/d2970e1d047f1bd31c31995c35450a7e5bfce3c0/cpp/src/arrow/filesystem/s3fs.cc#L1394-L1400

We've run into issues where the request rate to s3 is too high when uploading form a bigger cluster and S3 errors.

```
OSError("When completing multiple part upload for key '***' in bucket '***': AWS Error SLOW_DOWN during CompleteMultipartUpload operation: Please reduce your request rate.")
```

We've tried alleviating this with different s3 bucket prefixes, but this didn't solve the problem completely.

Would you be open to expose an option that makes the part size configurable so that we can configure the chunk size?

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.