cockroachdb / cockroachdb/cockroach
roachtest: add fault injection testing for imports on external storage
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
[Technical advisory a161317](https://www.cockroachlabs.com/docs/advisories/a161317) (silent data loss in AVRO OCF import, [#161317](https://github.com/cockroachdb/cockroach/issues/161317)) went undetected for ~5-6 years partly because there were no tests exercising import behavior under storage read errors.
CockroachDB already has a fault injection mechanism for external storage in `pkg/cloud/faulty/`, which wraps any `cloud.ExternalStorage` to inject probabilistic failures at the operation level (e.g., `ReadFile`, `List`) and I/O level (e.g., mid-read errors). It's used today by `pkg/backup/flaky_storage_test.go` via `base.TestingKnobs.CloudStorageKnobs`.
However, this mechanism is only accessible through `base.TestingKnobs`, so it cannot be used by roachtests which run against real cluster processes. We should expose it in a way that roachtests can activate it (e.g., via a cluster setting or similar mechanism), then add an import roachtest that:
- Enables fault injection on external storage reads
- Runs an import and verifies it either completes correctly (after retries) or fails with a clear error
- Confirms that silent data loss does not occur under storage faults
Jira issue: CRDB-60826
Epic CRDB-62881
Contributor guide
Assessment
This issue has not been assessed yet.