[Python/C++] Make upload size per part configurable when uploading to S3
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Currently, the size of every part in a multipart upload is hard-coded here: https://github.com/apache/arrow/blob/d2970e1d047f1bd31c31995c35450a7e5bfce3c0/cpp/src/arrow/filesystem/s3fs.cc#L1394-L1400
We've run into issues where the request rate to s3 is too high when uploading form a bigger cluster and S3 errors.
```
OSError("When completing multiple part upload for key '***' in bucket '***': AWS Error SLOW_DOWN during CompleteMultipartUpload operation: Please reduce your request rate.")
```
We've tried alleviating this with different s3 bucket prefixes, but this didn't solve the problem completely.
Would you be open to expose an option that makes the part size configurable so that we can configure the chunk size?
### Component(s)
C++, Python
Contributor guide
Assessment
This issue has not been assessed yet.