Repeated on-disk SST corruption and 0‑length files on GCP Hyperdisk Balanced (Pebble 1df7d4aae653)

### Pebble version
v0.0.0-20251103183323-1df7d4aae653 

### Environment
- Cloud: Google Cloud Platform (GCP)
- Storage: Hyperdisk Balanced zonal volumes, used as the primary data disk for the Pebble database
- Hyperdisk characteristics (high level):
- Next‑generation block storage with provisioned IOPS and throughput decoupled from VM size
- General‑purpose “Balanced” tier, up to 160k IOPS and ~2.4 GB/s throughput, targeted at stateful workloads such as databases
- Volumes are attached as standard block devices and formatted with a local filesystem (no network filesystems involved)
- Workload: custom database / indexer using Pebble as the storage engine, running continuously under typical production load (steady writes + background compactions)

### Summary

Between 2025‑12‑16 and 2026‑01‑20 we have observed 5 occurrences of on‑disk corruption affecting Pebble SSTables on the same GCP environment.

The symptoms are either:
- SSTable files with a size of 0 bytes, or
- SSTable files where Pebble reports a  crc32c checksum mismatch  when reading a data block during compaction.

The process does not crash at write time, and corruption is only detected later when the SSTable is read during compaction or iteration.

### Representative error

Example from the most recent incident:

```log
2026/01/19 21:05:04 on-disk corruption: /data/db/001333.sst; bounds: [\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x02\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04RuT\xaa\xb0,\x1c\xa9\xa2\xed\xb0\xfcK0\xc0\xbd\x19f\x063\x05\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe6\x078, \x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x02\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04Y\xeb3l\xd9\x94F\xe99\xed\x1d\xedE\W\xcc\xc4\x04\xfc_\x05\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x1c\x1dd]; details: pebble: file 001333: block 3339138/4918: crc32c checksum mismatch 0 != e4930ac7
(1) path: /data/db/001333.sst
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/sstable/block/block.go:523
| [...repeated from below...]
Wraps: (3) pebble: file 001333
Wraps: (4) forced error mark
| "pebble: corruption"
| github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (5) attached stack trace
-- stack trace:
| github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/internal/base/error.go:35
| github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/sstable/block/block.go:192
| github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/sstable/block/block.go:521
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/sstable/block/block.go:407
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/sstable/reader.go:394
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/sstable/reader_iter_single_lvl.go:507
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/sstable/reader_iter_single_lvl.go:1513
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/sstable/reader_iter_single_lvl.go:1397
| github.com/cockroachdb/pebble.(*levelIter).Next
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/level_iter.go:763
| github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/merging_iter.go:541
| github.com/cockroachdb/pebble.(*mergingIter).Next
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/merging_iter.go:1189
| github.com/cockroachdb/pebble/internal/compact.(*Iter).iterNext
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/internal/compact/iterator.go:710
| github.com/cockroachdb/pebble/internal/compact.(*Iter).nextInStripeHelper
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/internal/compact/iterator.go:753
| github.com/cockroachdb/pebble/internal/compact.(*Iter).nextInStripe
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/internal/compact/iterator.go:744
| github.com/cockroachdb/pebble/internal/compact.(*Iter).setNext
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/internal/compact/iterator.go:848
| github.com/cockroachdb/pebble/internal/compact.(*Iter).Next
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/internal/compact/iterator.go:604
| github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/internal/compact/run.go:267
| github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/internal/compact/run.go:200
| github.com/cockroachdb/pebble.(*DB).compactAndWrite
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:3473
| github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:3283
| github.com/cockroachdb/pebble.(*DB).runCompaction
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:3265
| github.com/cockroachdb/pebble.(*DB).compact1
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:2724
| github.com/cockroachdb/pebble.(*tableCompaction).Execute
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:386
| github.com/cockroachdb/pebble.(*DB).compact.func1.1
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:2592
| github.com/cockroachdb/pebble.(*DB).compact.func1
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:2621
| runtime/pprof.Do
| /usr/local/go/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble.(*DB).compact
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:2586
| github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| /go/pkg/mod/github.com/cockroachdb/pebble@v0.0.0-20251103183323-1df7d4aae653/compaction.go:2112
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1693
Wraps: (6) block 3339138/4918: crc32c checksum mismatch 0 != e4930ac7
Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.leafError
```

### Questions for Pebble maintainers

1. Are there any known issues in this Pebble revision (or nearby) that could lead to:
- writing an SSTable and leaving it 0‑length, or
- an SSTable being fsync’d but later failing checksum validation with  0  as the computed crc?
2. Are there recommended additional integrity checks or invariants we could enable (metrics, debug options, env hooks) to help differentiate between:
- genuine underlying storage corruption, vs
- a possible Pebble bug in SSTable writing, checksum calculation, or block caching?
3. Would upgrading to a newer Pebble version be advisable for any recent fixes related to corruption reporting or SSTable handling? If so, is there a specific minimum version you would recommend?
4. Is there any guidance for running Pebble on GCP Hyperdisk Balanced specifically (vs classic Persistent Disk SSD), e.g. around barriers, write‑ordering, or alignment assumptions?

Jira issue: PEBBLE-1329