cockroachdb / cockroachdb/cockroach

roachtest: add fault injection testing for imports on external storage

Open
#164,466 0 comments 0 reactions 0 assignees View on GitHub
A-import C-enhancement O-postmortem P-2 T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

[Technical advisory a161317](https://www.cockroachlabs.com/docs/advisories/a161317) (silent data loss in AVRO OCF import, [#161317](https://github.com/cockroachdb/cockroach/issues/161317)) went undetected for ~5-6 years partly because there were no tests exercising import behavior under storage read errors.

CockroachDB already has a fault injection mechanism for external storage in `pkg/cloud/faulty/`, which wraps any `cloud.ExternalStorage` to inject probabilistic failures at the operation level (e.g., `ReadFile`, `List`) and I/O level (e.g., mid-read errors). It's used today by `pkg/backup/flaky_storage_test.go` via `base.TestingKnobs.CloudStorageKnobs`.

However, this mechanism is only accessible through `base.TestingKnobs`, so it cannot be used by roachtests which run against real cluster processes. We should expose it in a way that roachtests can activate it (e.g., via a cluster setting or similar mechanism), then add an import roachtest that:
- Enables fault injection on external storage reads
- Runs an import and verifies it either completes correctly (after retries) or fails with a clear error
- Confirms that silent data loss does not occur under storage faults

Jira issue: CRDB-60826

Epic CRDB-62881

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.