cockroachdb / cockroachdb/cockroach

Sentry: pebble.go:1066: log.Fatal: local corruption detected: pebble: file 140425: block 2091681/14196: × checksum mismatch 844921e3 != e9a9ec22 (1) attached stack trace -- stack trace: | github....

Open
#159,498 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.2 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7112488685/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7112488685/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1066: log.Fatal: local corruption detected: pebble: file 140425: block 2091681/14196: × checksum mismatch 844921e3 != e9a9ec22
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1066
| github.com/cockroachdb/pebble.TeeEventListener.func2
| external/com_github_cockroachdb_pebble/event.go:994
| github.com/cockroachdb/pebble.(*DB).reportFileCorruption
| external/com_github_cockroachdb_pebble/event.go:1194
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1153
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:412
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:510
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:329
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:498
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1356
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1252
| github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).nextPos
| external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:646
| github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).Next
| external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:478
| github.com/cockroachdb/pebble.(*levelIter).Next
| external/com_github_cockroachdb_pebble/level_iter.go:753
| github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| external/com_github_cockroachdb_pebble/merging_iter.go:541
| github.com/cockroachdb/pebble.(*mergingIter).Next
| external/com_github_cockroachdb_pebble/merging_iter.go:1189
| github.com/cockroachdb/pebble.(*lazyCombinedIter).Next
| external/com_github_cockroachdb_pebble/range_keys.go:663
| github.com/cockroachdb/pebble.(*Iterator).nextUserKey
| external/com_github_cockroachdb_pebble/iterator.go:763
| github.com/cockroachdb/pebble.(*Iterator).nextWithLimit
| external/com_github_cockroachdb_pebble/iterator.go:1936
| github.com/cockroachdb/pebble.(*Iterator).Next
| external/com_github_cockroachdb_pebble/iterator.go:1701
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).Next
| pkg/storage/pebble_iterator.go:440
| github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).Next
| pkg/storage/intent_interleaving_iter.go:850
| github.com/cockroachdb/cockroach/pkg/storage.computeStatsForIterWithVisitors
| pkg/storage/mvcc.go:7399
| github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors
| pkg/storage/mvcc.go:7351
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors
| pkg/kv/kvserver/rditer/stats.go:90
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors
| pkg/kv/kvserver/rditer/stats.go:52
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.CalcReplicaDigest
| pkg/kv/kvserver/replica_consistency.go:626
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2
| pkg/kv/kvserver/replica_consistency.go:749
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:500
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) secondary error attachment
| pebble: file 140425: block 2091681/14196: × checksum mismatch 844921e3 != e9a9ec22
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:572
| | [...repeated from below...]
| Wraps: (3) pebble: file 140425
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:178
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:570
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:507
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:329
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:498
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1356
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1252
| | github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).nextPos
| | external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:646
| | github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).Next
| | external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:478
| | github.com/cockroachdb/pebble.(*levelIter).Next
| | external/com_github_cockroachdb_pebble/level_iter.go:753
| | github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| | external/com_github_cockroachdb_pebble/merging_iter.go:541
| | github.com/cockroachdb/pebble.(*mergingIter).Next
| | external/com_github_cockroachdb_pebble/merging_iter.go:1189
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).Next
| | external/com_github_cockroachdb_pebble/range_keys.go:663
| | github.com/cockroachdb/pebble.(*Iterator).nextUserKey
| | external/com_github_cockroachdb_pebble/iterator.go:763
| | github.com/cockroachdb/pebble.(*Iterator).nextWithLimit
| | external/com_github_cockroachdb_pebble/iterator.go:1936
| | github.com/cockroachdb/pebble.(*Iterator).Next
| | external/com_github_cockroachdb_pebble/iterator.go:1701
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).Next
| | pkg/storage/pebble_iterator.go:440
| | github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).Next
| | pkg/storage/intent_interleaving_iter.go:850
| | github.com/cockroachdb/cockroach/pkg/storage.computeStatsForIterWithVisitors
| | pkg/storage/mvcc.go:7399
| | github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors
| | pkg/storage/mvcc.go:7351
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors
| | pkg/kv/kvserver/rditer/stats.go:90
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors
| | pkg/kv/kvserver/rditer/stats.go:52
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.CalcReplicaDigest
| | pkg/kv/kvserver/replica_consistency.go:626
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2
| | pkg/kv/kvserver/replica_consistency.go:749
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | pkg/util/stop/stopper.go:500
| | runtime.goexit
| | src/runtime/asm_amd64.s:1700
| Wraps: (6) block 2091681/14196: × checksum mismatc...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L499-L501](pkg/util/stop/stopper.go#L499-L501)
[pkg/kv/kvserver/replica_consistency.go#L748-L750](pkg/kv/kvserver/replica_consistency.go#L748-L750)
[pkg/kv/kvserver/replica_consistency.go#L625-L627](pkg/kv/kvserver/replica_consistency.go#L625-L627)
[pkg/kv/kvserver/rditer/stats.go#L51-L53](pkg/kv/kvserver/rditer/stats.go#L51-L53)
[pkg/kv/kvserver/rditer/stats.go#L89-L91](pkg/kv/kvserver/rditer/stats.go#L89-L91)
[pkg/storage/mvcc.go#L7350-L7352](pkg/storage/mvcc.go#L7350-L7352)
[pkg/storage/mvcc.go#L7398-L7400](pkg/storage/mvcc.go#L7398-L7400)
[pkg/storage/intent_interleaving_iter.go#L849-L851](pkg/storage/intent_interleaving_iter.go#L849-L851)
[pkg/storage/pebble_iterator.go#L439-L441](pkg/storage/pebble_iterator.go#L439-L441)
[external/com_github_cockroachdb_pebble/iterator.go#L1700-L1702](external/com_github_cockroachdb_pebble/iterator.go#L1700-L1702)
[external/com_github_cockroachdb_pebble/iterator.go#L1935-L1937](external/com_github_cockroachdb_pebble/iterator.go#L1935-L1937)
[external/com_github_cockroachdb_pebble/iterator.go#L762-L764](external/com_github_cockroachdb_pebble/iterator.go#L762-L764)
[external/com_github_cockroachdb_pebble/range_keys.go#L662-L664](external/com_github_cockroachdb_pebble/range_keys.go#L662-L664)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1188-L1190](external/com_github_cockroachdb_pebble/merging_iter.go#L1188-L1190)
[external/com_github_cockroachdb_pebble/merging_iter.go#L540-L542](external/com_github_cockroachdb_pebble/merging_iter.go#L540-L542)
[external/com_github_cockroachdb_pebble/level_iter.go#L752-L754](external/com_github_cockroachdb_pebble/level_iter.go#L752-L754)
[external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L477-L479](external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L477-L479)
[external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L645-L647](external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L645-L647)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1251-L1253](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1251-L1253)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1355-L1357](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1355-L1357)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L497-L499](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L497-L499)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L328-L330](external/com_github_cockroachdb_pebble/sstable/reader.go#L328-L330)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511](external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413](external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413)
[external/com_github_cockroachdb_pebble/event.go#L1152-L1154](external/com_github_cockroachdb_pebble/event.go#L1152-L1154)
[external/com_github_cockroachdb_pebble/event.go#L1193-L1195](external/com_github_cockroachdb_pebble/event.go#L1193-L1195)
[external/com_github_cockroachdb_pebble/event.go#L993-L995](external/com_github_cockroachdb_pebble/event.go#L993-L995)
[pkg/storage/pebble.go#L1065-L1067](pkg/storage/pebble.go#L1065-L1067)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/kv/kvserver/replica_consistency.go in pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2 at line 749
pkg/kv/kvserver/replica_consistency.go in pkg/kv/kvserver.CalcReplicaDigest at line 626
pkg/kv/kvserver/rditer/stats.go in pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors at line 52
pkg/kv/kvserver/rditer/stats.go in pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors at line 90
pkg/storage/mvcc.go in pkg/storage.ComputeStatsWithVisitors at line 7351
pkg/storage/mvcc.go in pkg/storage.computeStatsForIterWithVisitors at line 7399
pkg/storage/intent_interleaving_iter.go in pkg/storage.(*intentInterleavingIter).Next at line 850
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).Next at line 440
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).Next at line 1701
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).nextWithLimit at line 1936
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).nextUserKey at line 763
external/com_github_cockroachdb_pebble/range_keys.go in github.com/cockroachdb/pebble.(*lazyCombinedIter).Next at line 663
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).Next at line 1189
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).nextEntry at line 541
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).Next at line 753
external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go in github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).Next at line 478
external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go in github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).nextPos at line 646
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next at line 1252
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward at line 1356
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock at line 498
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock at line 329
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 510
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 412
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1153
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportFileCorruption at line 1194
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func2 at line 994
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1066
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.2 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.2 |
| Cockroach SHA | 0780d511bd527bfd9d02f4c7ca5ecc19115d9f5e |
| # of CPUs | 8 |
| # of Goroutines | 979 |

Jira issue: CRDB-57895

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.