cockroachdb / cockroachdb/cockroach
backup: retry writing metadata files fails when delete protection is enabled
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
For external storage providers, rewriting an existing blob object is implemented as delete-then-rewrite. Backup was designed on the assumption that it does not require delete permission. As a result, for customers who have **delete protection** enabled on their bucket, a backup that retries and attempts to rewrite an already-written object fails on the missing delete permission.
**Impact**
- Only affects customers with **delete protection** enabled on their backup bucket.
- Only bites when a backup fails **during the final metadata-writing step** (manifest, descriptors, index, etc.).
**Failure scenario**
1. Backup writes the manifest, descriptors, and other metadata files.
2. Writing the index fails (or some other late-stage error occurs).
3. The job retries and reaches the concluding step again.
4. It attempts to rewrite the already-present metadata files → delete-then-rewrite → fails on missing delete permission.
**Proposed options**
1. **Never write the same file twice (unique/timestamp-suffixed names).** Give each metadata file a unique name so nothing is ever rewritten. The index is the last file written, so on recovery we can **reverse-list the `backup metadata`-prefixed objects and take the latest** — the index's presence implies all prior writes succeeded, so the index itself doesn't need to track timestamps.
2. **Swallow the error after probing existence.** On a write/close error, probe the bucket; if the object already exists, drop the error and continue. Rationale: if the key exists, a prior write to it succeeded, and the retry rewrites byte-identical content (fileless SST, descriptorless SST, metadata, index — none expected to change). Lowest effort, but the shakiest — it makes assumptions that could break later. Better applied case-by-case to writers we can prove idempotency for (e.g. the descriptor-list writer only ever writes the same set of descriptors) than as a blanket swallow.
3. **Do nothing.** These metadata files are tiny and only fail to write when some other bug is present; subsequent scheduled backup jobs will generally succeed on their own.
Jira issue: CRDB-65382
Contributor guide
Assessment
This issue has not been assessed yet.