cockroachdb / cockroachdb/cockroach

backupccl: more compatible use of S3 delimiter

Open
#100,123 2 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-enhancement T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

Backup & restore use `data/` as a delimiter to accelerate file listing on S3. This is good!

However, it turns out that S3-“compatible” services might not accept a multi-character delimiter. AWS [**defines**](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) a delimiter as a “character”; their service **accepts** multi characters.

While we only officially support and test against “actual”-AWS-from-Amazon, there are many third-party S3-“compatible” services that people use.

- https://github.com/cockroachdb/cockroach/issues/97864
- https://github.com/cockroachdb/cockroach/issues/69165

Seems like MinIO, Alibaba and Tencent S3 services seem to struggle with this delimiter.

**Describe the solution you'd like**

Choose a delimiter that gives us compatibility with AWS docs and more third-party S3’s. [See internal conversation](https://cockroachlabs.slack.com/archives/CJXBR3693/p1677704478141109?thread_ts=1677704068.755369&cid=CJXBR3693).

Jira issue: CRDB-26265

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.