cockroachdb / cockroachdb/cockroach
online restore: pkg/cloud context cancellation errors can result in false positive replica corruption errors
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
One of these was fixed in https://github.com/cockroachdb/cockroach/pull/123313
> The code in s3_storage.go is now also used by Pebble's disaggregated storage, and context cancellation gets propagated to the s3 library (as desired). There are callers in CockroachDB that use the returned error and decide whether it represents corruption, and need to realize that an error due to context cancellation is not corruption. The immediate motivator for this change is kvpb.MaybeWrapReplicaCorruptionError, used when the abort span read returns an error. It was mistaking these s3 errors as corruption
There may be others lurking:
- s3_storage.go has a couple of places where it injects its own timeouts. These are marked with TODO(sumeer) and should be fixed.
- Other blob storage libraries may also have such behavior that needs auditing.
- We need more testing to increase confidence that we have addressed all of such bugs, by injecting long delays.
- If such a bug does slip through and gets tickled in production, we need a runbook entry that instructs the operator to look at the error message, notice that it is a blob storage timeout, delete the CRITICAL_ALERT file, and restart the node (since this is unlikely to recur, and we don't want loss of quorum due to coordinated failures caused by the blob storage being transiently slow).
Jira issue: CRDB-38369
Epic CRDB-40359
Contributor guide
Assessment
This issue has not been assessed yet.