cockroachdb / cockroachdb/cockroach

online restore: pkg/cloud context cancellation errors can result in false positive replica corruption errors

Open
#123,478 2 comments 0 reactions 0 assignees View on GitHub
A-disaster-recovery A-storage C-bug T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

One of these was fixed in https://github.com/cockroachdb/cockroach/pull/123313
> The code in s3_storage.go is now also used by Pebble's disaggregated storage, and context cancellation gets propagated to the s3 library (as desired). There are callers in CockroachDB that use the returned error and decide whether it represents corruption, and need to realize that an error due to context cancellation is not corruption. The immediate motivator for this change is kvpb.MaybeWrapReplicaCorruptionError, used when the abort span read returns an error. It was mistaking these s3 errors as corruption

There may be others lurking:
- s3_storage.go has a couple of places where it injects its own timeouts. These are marked with TODO(sumeer) and should be fixed.
- Other blob storage libraries may also have such behavior that needs auditing.
- We need more testing to increase confidence that we have addressed all of such bugs, by injecting long delays.
- If such a bug does slip through and gets tickled in production, we need a runbook entry that instructs the operator to look at the error message, notice that it is a blob storage timeout, delete the CRITICAL_ALERT file, and restart the node (since this is unlikely to recur, and we don't want loss of quorum due to coordinated failures caused by the blob storage being transiently slow).

Jira issue: CRDB-38369

Epic CRDB-40359

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.