cockroachdb / cockroachdb/cockroach
backupccl: more compatible use of S3 delimiter
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Backup & restore use `data/` as a delimiter to accelerate file listing on S3. This is good!
However, it turns out that S3-“compatible” services might not accept a multi-character delimiter. AWS [**defines**](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) a delimiter as a “character”; their service **accepts** multi characters.
While we only officially support and test against “actual”-AWS-from-Amazon, there are many third-party S3-“compatible” services that people use.
- https://github.com/cockroachdb/cockroach/issues/97864
- https://github.com/cockroachdb/cockroach/issues/69165
Seems like MinIO, Alibaba and Tencent S3 services seem to struggle with this delimiter.
**Describe the solution you'd like**
Choose a delimiter that gives us compatibility with AWS docs and more third-party S3’s. [See internal conversation](https://cockroachlabs.slack.com/archives/CJXBR3693/p1677704478141109?thread_ts=1677704068.755369&cid=CJXBR3693).
Jira issue: CRDB-26265
Contributor guide
Assessment
This issue has not been assessed yet.