cockroachdb / cockroachdb/cockroach
roachtest/roachprod: capture forensic state from local-SSD clusters before teardown
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When a roachtest detects a catastrophic issue (e.g. on-disk corruption, panic), it calls `roachprod.CreateSnapshot` to capture an "invariant-problem-*" snapshot of the cluster for post-mortem analysis. This works for clusters using persistent disks but **does not capture any state on clusters using local SSDs**.
The root cause is that GCE has no API to snapshot local SSDs — they are ephemeral hardware physically attached to the host machine, not standalone cloud resources. After #168544 fixed `ListVolumes` to no longer crash on local-SSD clusters, the snapshot operation now exits cleanly with:
node N does not have any non-bootable persistent volumes attached
But the cluster is still torn down with its forensic state intact on the local SSDs, where it is destroyed.
Motivating incident: `tpccbench/nodes=6/cpu=16/multi-az` on release-26.2 (TeamCity build 21312957) hit a Pebble SST CRC32c checksum mismatch on n6's local SSD (#168434). The invariant-problem snapshot attempt failed — originally with the `ListVolumes` crash from #168544; after that fix, with the cleaner "no persistent volumes" error. Either way, the corrupted state was destroyed at teardown.
## Suggested Solution
- attaching a temporary persistent disk to the VM
- block-copying each local SSD onto it via dd to preserve the exact bytes
- then snapshotting the PD
### Additional Context
- Parent issue: #168544 (fixed the `ListVolumes` crash; explicitly noted this gap as out-of-scope).
- Original incident: #168434 (Pebble SST corruption on local SSD)
Jira issue: CRDB-64161
Contributor guide
Assessment
This issue has not been assessed yet.