S3FileSystem Cloudflare R2 workaround limits maximum filesize upload for default AWS S3
Open
Component: C++
Type: enhancement
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
This workaround for Cloudflare R2 storage sets the multipart upload size to a non-configurable 10MB, which results in a maximum filesize of 98GB for uploading. This is a limitation that should not spill over to the more common AWS S3 and Minio S3 options.
https://github.com/apache/arrow/blob/640c10191a51f6d0f408c72f45dbf5d94ec0b9d7/cpp/src/arrow/filesystem/s3fs.cc#L1539-L1546
One possible solution: Make this configurable via some environment variable or API, and require Cloudflare R2 users to set it.
### Component(s)
C++
Contributor guide
Assessment
This issue has not been assessed yet.