Sentry: pebble.go:1373: log.Dev.Fatal: local corruption detected: pebble: file 000156: block 3236448/23435: × checksum mismatch 688eda2e != 58ac6eaa (1) attached stack trace -- stack trace: | git...

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7477986900/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7477986900/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1373: log.Dev.Fatal: local corruption detected: pebble: file 000156: block 3236448/23435: × checksum mismatch 688eda2e != 58ac6eaa
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1373
| github.com/cockroachdb/pebble.TeeEventListener.func6
| external/com_github_cockroachdb_pebble/event.go:1243
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1430
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:323
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:461
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:391
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:508
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1654
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).nextInternal
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1523
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1461
| github.com/cockroachdb/pebble.(*levelIter).internalNext
| external/com_github_cockroachdb_pebble/level_iter.go:862
| github.com/cockroachdb/pebble.(*levelIter).Next
| external/com_github_cockroachdb_pebble/level_iter.go:827
| github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| external/com_github_cockroachdb_pebble/merging_iter.go:560
| github.com/cockroachdb/pebble.(*mergingIter).Next
| external/com_github_cockroachdb_pebble/merging_iter.go:1251
| github.com/cockroachdb/pebble.(*lazyCombinedIter).Next
| external/com_github_cockroachdb_pebble/range_keys.go:662
| github.com/cockroachdb/pebble.(*Iterator).nextUserKey
| external/com_github_cockroachdb_pebble/iterator.go:790
| github.com/cockroachdb/pebble.(*Iterator).nextWithLimit
| external/com_github_cockroachdb_pebble/iterator.go:2024
| github.com/cockroachdb/pebble.(*Iterator).Next
| external/com_github_cockroachdb_pebble/iterator.go:1772
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).Next
| pkg/storage/pebble_iterator.go:437
| github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).Next
| pkg/storage/intent_interleaving_iter.go:851
| github.com/cockroachdb/cockroach/pkg/storage.computeStatsForIterWithVisitors
| pkg/storage/mvcc.go:7624
| github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors.func1
| pkg/storage/mvcc.go:7506
| github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors
| pkg/storage/mvcc.go:7523
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors
| pkg/kv/kvserver/rditer/stats.go:110
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors
| pkg/kv/kvserver/rditer/stats.go:66
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.ComputeStatsForRange
| pkg/kv/kvserver/rditer/stats.go:26
| github.com/cockroachdb/cockroach/pkg/cli.worker
| debug_check_store.go:128
| github.com/cockroachdb/cockroach/pkg/cli.checkStoreRangeStats.func1
| debug_check_store.go:159
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (2) secondary error attachment
| pebble: file 000156: block 3236448/23435: × checksum mismatch 688eda2e != 58ac6eaa
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:532
| | [...repeated from below...]
| Wraps: (3) pebble: file 000156
| Wraps: (4) . bit flip found: byte index 2916. got: 0x16. want: 0x1e.
| Wraps: (5) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (6) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:193
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:530
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:458
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:391
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:508
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1654
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).nextInternal
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1523
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1461
| | github.com/cockroachdb/pebble.(*levelIter).internalNext
| | external/com_github_cockroachdb_pebble/level_iter.go:862
| | github.com/cockroachdb/pebble.(*levelIter).Next
| | external/com_github_cockroachdb_pebble/level_iter.go:827
| | github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| | external/com_github_cockroachdb_pebble/merging_iter.go:560
| | github.com/cockroachdb/pebble.(*mergingIter).Next
| | external/com_github_cockroachdb_pebble/merging_iter.go:1251
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).Next
| | external/com_github_cockroachdb_pebble/range_keys.go:662
| | github.com/cockroachdb/pebble.(*Iterator).nextUserKey
| | external/com_github_cockroachdb_pebble/iterator.go:790
| | github.com/cockroachdb/pebble.(*Iterator).nextWithLimit
| | external/com_github_cockroachdb_pebble/iterator.go:2024
| | github.com/cockroachdb/pebble.(*Iterator).Next
| | external/com_github_cockroachdb_pebble/iterator.go:1772
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).Next
| | pkg/storage/pebble_iterator.go:437
| | github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).Next
| | pkg/storage/intent_interleaving_iter.go:851
| | github.com/cockroachdb/cockroach/pkg/storage.computeStatsForIterWithVisitors
| | pkg/storage/mvcc.go:7624
| | github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors.func1
| | pkg/storage/mvcc.go:7506
| | github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors
| | pkg/storage/mvcc.go:7523
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors
| | pkg/kv/kvserver/rditer/stats.go:110
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors
| | pkg/kv/kvserver/rditer/stats.go:66
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.ComputeStatsForRange
| | pkg/kv/kvserver/rditer/stats.go:26
| | github.com/cockroachdb/cockroach/pkg/cli.worker
| | debug_check_store.go:128
| | github.com/cockroachdb/cockroach/pkg/cli.checkStoreRangeStats.func1
| | debug_check_store.go:159
| | golang.org/x/sync/errgroup.(*Group).Go....
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1692-L1694](src/runtime/asm_amd64.s#L1692-L1694)
[external/org_golang_x_sync/errgroup/errgroup.go#L92-L94](external/org_golang_x_sync/errgroup/errgroup.go#L92-L94)
[debug_check_store.go#L158-L160](debug_check_store.go#L158-L160)
[debug_check_store.go#L127-L129](debug_check_store.go#L127-L129)
[pkg/kv/kvserver/rditer/stats.go#L25-L27](pkg/kv/kvserver/rditer/stats.go#L25-L27)
[pkg/kv/kvserver/rditer/stats.go#L65-L67](pkg/kv/kvserver/rditer/stats.go#L65-L67)
[pkg/kv/kvserver/rditer/stats.go#L109-L111](pkg/kv/kvserver/rditer/stats.go#L109-L111)
[pkg/storage/mvcc.go#L7522-L7524](pkg/storage/mvcc.go#L7522-L7524)
[pkg/storage/mvcc.go#L7505-L7507](pkg/storage/mvcc.go#L7505-L7507)
[pkg/storage/mvcc.go#L7623-L7625](pkg/storage/mvcc.go#L7623-L7625)
[pkg/storage/intent_interleaving_iter.go#L850-L852](pkg/storage/intent_interleaving_iter.go#L850-L852)
[pkg/storage/pebble_iterator.go#L436-L438](pkg/storage/pebble_iterator.go#L436-L438)
[external/com_github_cockroachdb_pebble/iterator.go#L1771-L1773](external/com_github_cockroachdb_pebble/iterator.go#L1771-L1773)
[external/com_github_cockroachdb_pebble/iterator.go#L2023-L2025](external/com_github_cockroachdb_pebble/iterator.go#L2023-L2025)
[external/com_github_cockroachdb_pebble/iterator.go#L789-L791](external/com_github_cockroachdb_pebble/iterator.go#L789-L791)
[external/com_github_cockroachdb_pebble/range_keys.go#L661-L663](external/com_github_cockroachdb_pebble/range_keys.go#L661-L663)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1250-L1252](external/com_github_cockroachdb_pebble/merging_iter.go#L1250-L1252)
[external/com_github_cockroachdb_pebble/merging_iter.go#L559-L561](external/com_github_cockroachdb_pebble/merging_iter.go#L559-L561)
[external/com_github_cockroachdb_pebble/level_iter.go#L826-L828](external/com_github_cockroachdb_pebble/level_iter.go#L826-L828)
[external/com_github_cockroachdb_pebble/level_iter.go#L861-L863](external/com_github_cockroachdb_pebble/level_iter.go#L861-L863)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1460-L1462](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1460-L1462)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1522-L1524](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1522-L1524)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1653-L1655](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1653-L1655)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L507-L509](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L507-L509)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L390-L392](external/com_github_cockroachdb_pebble/sstable/reader.go#L390-L392)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L460-L462](external/com_github_cockroachdb_pebble/sstable/block/block.go#L460-L462)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L322-L324](external/com_github_cockroachdb_pebble/sstable/block/block.go#L322-L324)
[external/com_github_cockroachdb_pebble/event.go#L1429-L1431](external/com_github_cockroachdb_pebble/event.go#L1429-L1431)
[external/com_github_cockroachdb_pebble/event.go#L1242-L1244](external/com_github_cockroachdb_pebble/event.go#L1242-L1244)
[pkg/storage/pebble.go#L1372-L1374](pkg/storage/pebble.go#L1372-L1374)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1693
external/org_golang_x_sync/errgroup/errgroup.go in golang.org/x/sync/errgroup.(*Group).Go.func1 at line 93
debug_check_store.go in pkg/cli.checkStoreRangeStats.func1 at line 159
debug_check_store.go in pkg/cli.worker at line 128
pkg/kv/kvserver/rditer/stats.go in pkg/kv/kvserver/rditer.ComputeStatsForRange at line 26
pkg/kv/kvserver/rditer/stats.go in pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors at line 66
pkg/kv/kvserver/rditer/stats.go in pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors at line 110
pkg/storage/mvcc.go in pkg/storage.ComputeStatsWithVisitors at line 7523
pkg/storage/mvcc.go in pkg/storage.ComputeStatsWithVisitors.func1 at line 7506
pkg/storage/mvcc.go in pkg/storage.computeStatsForIterWithVisitors at line 7624
pkg/storage/intent_interleaving_iter.go in pkg/storage.(*intentInterleavingIter).Next at line 851
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).Next at line 437
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).Next at line 1772
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).nextWithLimit at line 2024
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).nextUserKey at line 790
external/com_github_cockroachdb_pebble/range_keys.go in github.com/cockroachdb/pebble.(*lazyCombinedIter).Next at line 662
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).Next at line 1251
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).nextEntry at line 560
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).Next at line 827
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).internalNext at line 862
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next at line 1461
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).nextInternal at line 1523
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward at line 1654
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock at line 508
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock at line 391
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 461
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 323
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1430
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1243
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1373
```

### Tags

| Tag | Value |
| --- | --- |
| Command | debug check-store |
| Environment | v26.1.1 |
| Go Version | go1.25.5 |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v26.1.1 |
| Cockroach SHA | 5abbbd24fcc2c7873fb0f985ccea68b7abaa6afe |
| # of CPUs | 12 |
| # of Goroutines | 33 |

Jira issue: CRDB-63899