cockroachdb / cockroachdb/cockroach

roachtest/roachprod: capture forensic state from local-SSD clusters before teardown

Open
#170,764 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement O-agent T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When a roachtest detects a catastrophic issue (e.g. on-disk corruption, panic), it calls `roachprod.CreateSnapshot` to capture an "invariant-problem-*" snapshot of the cluster for post-mortem analysis. This works for clusters using persistent disks but **does not capture any state on clusters using local SSDs**.

The root cause is that GCE has no API to snapshot local SSDs — they are ephemeral hardware physically attached to the host machine, not standalone cloud resources. After #168544 fixed `ListVolumes` to no longer crash on local-SSD clusters, the snapshot operation now exits cleanly with:

node N does not have any non-bootable persistent volumes attached

But the cluster is still torn down with its forensic state intact on the local SSDs, where it is destroyed.

Motivating incident: `tpccbench/nodes=6/cpu=16/multi-az` on release-26.2 (TeamCity build 21312957) hit a Pebble SST CRC32c checksum mismatch on n6's local SSD (#168434). The invariant-problem snapshot attempt failed — originally with the `ListVolumes` crash from #168544; after that fix, with the cleaner "no persistent volumes" error. Either way, the corrupted state was destroyed at teardown.

## Suggested Solution
- attaching a temporary persistent disk to the VM
- block-copying each local SSD onto it via dd to preserve the exact bytes
- then snapshotting the PD

### Additional Context

- Parent issue: #168544 (fixed the `ListVolumes` crash; explicitly noted this gap as out-of-scope).
- Original incident: #168434 (Pebble SST corruption on local SSD)

Jira issue: CRDB-64161

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.