cockroachdb / cockroachdb/cockroach

backup: retry writing metadata files fails when delete protection is enabled

Open
#172,217 2 comments 0 reactions 1 assignee Claimed by @kev-cao View on GitHub
A-disaster-recovery branch-master C-bug T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

For external storage providers, rewriting an existing blob object is implemented as delete-then-rewrite. Backup was designed on the assumption that it does not require delete permission. As a result, for customers who have **delete protection** enabled on their bucket, a backup that retries and attempts to rewrite an already-written object fails on the missing delete permission.

**Impact**

- Only affects customers with **delete protection** enabled on their backup bucket.
- Only bites when a backup fails **during the final metadata-writing step** (manifest, descriptors, index, etc.).

**Failure scenario**

1. Backup writes the manifest, descriptors, and other metadata files.
2. Writing the index fails (or some other late-stage error occurs).
3. The job retries and reaches the concluding step again.
4. It attempts to rewrite the already-present metadata files → delete-then-rewrite → fails on missing delete permission.

**Proposed options**

1. **Never write the same file twice (unique/timestamp-suffixed names).** Give each metadata file a unique name so nothing is ever rewritten. The index is the last file written, so on recovery we can **reverse-list the `backup metadata`-prefixed objects and take the latest** — the index's presence implies all prior writes succeeded, so the index itself doesn't need to track timestamps.
2. **Swallow the error after probing existence.** On a write/close error, probe the bucket; if the object already exists, drop the error and continue. Rationale: if the key exists, a prior write to it succeeded, and the retry rewrites byte-identical content (fileless SST, descriptorless SST, metadata, index — none expected to change). Lowest effort, but the shakiest — it makes assumptions that could break later. Better applied case-by-case to writers we can prove idempotency for (e.g. the descriptor-list writer only ever writes the same set of descriptors) than as a blanket swallow.
3. **Do nothing.** These metadata files are tiny and only fail to write when some other bug is present; subsequent scheduled backup jobs will generally succeed on their own.

Jira issue: CRDB-65382

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.