cockroachdb / cockroachdb/cockroach

Sentry: pebble.go:1066: log.Fatal: local corruption detected: pebble: file 096765: block 23959913/29793: × checksum mismatch 79206e50 != 3aecca17 (1) attached stack trace -- stack trace: | github...

Open
#159,816 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.2 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7121990985/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7121990985/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1066: log.Fatal: local corruption detected: pebble: file 096765: block 23959913/29793: × checksum mismatch 79206e50 != 3aecca17
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1066
| github.com/cockroachdb/pebble.TeeEventListener.func2
| external/com_github_cockroachdb_pebble/event.go:994
| github.com/cockroachdb/pebble.(*DB).reportFileCorruption
| external/com_github_cockroachdb_pebble/event.go:1194
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1153
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:412
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:510
| github.com/cockroachdb/pebble/sstable.(*Reader).readIndexBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:313
| github.com/cockroachdb/pebble/sstable.(*Reader).readTopLevelIndexBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:305
| github.com/cockroachdb/pebble/sstable.newColumnBlockSingleLevelIterator
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:219
| github.com/cockroachdb/pebble/sstable.(*Reader).newPointIter
| external/com_github_cockroachdb_pebble/sstable/reader.go:134
| github.com/cockroachdb/pebble/sstable.(*Reader).NewPointIter
| external/com_github_cockroachdb_pebble/sstable/reader.go:97
| github.com/cockroachdb/pebble.(*fileCacheHandle).newPointIter
| external/com_github_cockroachdb_pebble/file_cache.go:686
| github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| external/com_github_cockroachdb_pebble/file_cache.go:566
| github.com/cockroachdb/pebble.(*levelIter).loadFile
| external/com_github_cockroachdb_pebble/level_iter.go:568
| github.com/cockroachdb/pebble.(*levelIter).skipEmptyFileBackward
| external/com_github_cockroachdb_pebble/level_iter.go:880
| github.com/cockroachdb/pebble.(*levelIter).Prev
| external/com_github_cockroachdb_pebble/level_iter.go:805
| github.com/cockroachdb/pebble.(*mergingIter).prevEntry
| external/com_github_cockroachdb_pebble/merging_iter.go:761
| github.com/cockroachdb/pebble.(*mergingIter).Prev
| external/com_github_cockroachdb_pebble/merging_iter.go:1291
| github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).prevPos
| external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:718
| github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).Prev
| external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:578
| github.com/cockroachdb/pebble.(*Iterator).findPrevEntry
| external/com_github_cockroachdb_pebble/iterator.go:1058
| github.com/cockroachdb/pebble.(*Iterator).PrevWithLimit
| external/com_github_cockroachdb_pebble/iterator.go:2089
| github.com/cockroachdb/pebble.(*Iterator).Prev
| external/com_github_cockroachdb_pebble/iterator.go:1993
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).Prev
| pkg/storage/pebble_iterator.go:611
| github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).Prev
| pkg/storage/intent_interleaving_iter.go:1276
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.(*gcIterator).fillTo
| pkg/kv/kvserver/gc/gc_iterator.go:160
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.(*gcIterator).peekAt
| pkg/kv/kvserver/gc/gc_iterator.go:120
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.(*gcIterator).state
| pkg/kv/kvserver/gc/gc_iterator.go:93
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.processReplicatedKeyRange.func1
| pkg/kv/kvserver/gc/gc.go:485
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.IterateMVCCReplicaKeySpans.func1
| pkg/kv/kvserver/rditer/replica_data_iter.go:526
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.IterateMVCCReplicaKeySpans
| pkg/kv/kvserver/rditer/replica_data_iter.go:529
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.processReplicatedKeyRange
| pkg/kv/kvserver/gc/gc.go:453
Wraps: (2) secondary error attachment
| pebble: file 096765: block 23959913/29793: × checksum mismatch 79206e50 != 3aecca17
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:572
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:507
| | github.com/cockroachdb/pebble/sstable.(*Reader).readIndexBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:313
| | github.com/cockroachdb/pebble/sstable.(*Reader).readTopLevelIndexBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:305
| | github.com/cockroachdb/pebble/sstable.newColumnBlockSingleLevelIterator
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:219
| | github.com/cockroachdb/pebble/sstable.(*Reader).newPointIter
| | external/com_github_cockroachdb_pebble/sstable/reader.go:134
| | github.com/cockroachdb/pebble/sstable.(*Reader).NewPointIter
| | external/com_github_cockroachdb_pebble/sstable/reader.go:97
| | github.com/cockroachdb/pebble.(*fileCacheHandle).newPointIter
| | external/com_github_cockroachdb_pebble/file_cache.go:686
| | github.com/cockroachdb/pebble.(*fileCacheHandle).newIters
| | external/com_github_cockroachdb_pebble/file_cache.go:566
| | github.com/cockroachdb/pebble.(*levelIter).loadFile
| | external/com_github_cockroachdb_pebble/level_iter.go:568
| | github.com/cockroachdb/pebble.(*levelIter).skipEmptyFileBackward
| | external/com_github_cockroachdb_pebble/level_iter.go:880
| | github.com/cockroachdb/pebble.(*levelIter).Prev
| | external/com_github_cockroachdb_pebble/level_iter.go:805
| | github.com/cockroachdb/pebble.(*mergingIter).prevEntry
| | external/com_github_cockroachdb_pebble/merging_iter.go:761
| | github.com/cockroachdb/pebble.(*mergingIter).Prev
| | external/com_github_cockroachdb_pebble/merging_iter.go:1291
| | github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).prevPos
| | external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:718
| | github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).Prev
| | external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:578
| | github.com/cockroachdb/pebble.(*Iterator).findPrevEntry
| | external/com_github_cockroachdb_pebble/iterator.go:1058
| | github.com/cockroachdb/pebble.(*Iterator).PrevWithLimit
| | external/com_github_cockroachdb_pebble/iterator.go:2089
| | github.com/cockroachdb/pebble.(*Iterator).Prev
| | external/com_github_cockroachdb_pebble/iterator.go:1993
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).Prev
| | pkg/storage/pebble_iterator.go:611
| | github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).Prev
| | pkg/storage/intent_interleaving_iter.go:1276
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.(*gcIterator).fillTo
| | pkg/kv/kvserver/gc/gc_iterator.go:160
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.(*gcIterator).peekAt
| | pkg/kv/kvserver/gc/gc_iterator.go:120
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.(*gcIterator).state
| | pkg/kv/kvserver/gc/gc_iterator.go:93
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc.processReplicatedKeyRange.func1
| | pkg/kv/kvserver/gc/gc.go:485
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.IterateMVCCReplicaKeySpans.func1
| | pkg/kv/kvserver/rditer/replica_data_iter.go:526
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.IterateMVCCReplicaKeySpans
| | pkg/kv/kvserver/rditer/r...
```

Stacktrace (expand for inline code snippets):

[pkg/kv/kvserver/gc/gc.go#L452-L454](pkg/kv/kvserver/gc/gc.go#L452-L454)
[pkg/kv/kvserver/rditer/replica_data_iter.go#L528-L530](pkg/kv/kvserver/rditer/replica_data_iter.go#L528-L530)
[pkg/kv/kvserver/rditer/replica_data_iter.go#L525-L527](pkg/kv/kvserver/rditer/replica_data_iter.go#L525-L527)
[pkg/kv/kvserver/gc/gc.go#L484-L486](pkg/kv/kvserver/gc/gc.go#L484-L486)
[pkg/kv/kvserver/gc/gc_iterator.go#L92-L94](pkg/kv/kvserver/gc/gc_iterator.go#L92-L94)
[pkg/kv/kvserver/gc/gc_iterator.go#L119-L121](pkg/kv/kvserver/gc/gc_iterator.go#L119-L121)
[pkg/kv/kvserver/gc/gc_iterator.go#L159-L161](pkg/kv/kvserver/gc/gc_iterator.go#L159-L161)
[pkg/storage/intent_interleaving_iter.go#L1275-L1277](pkg/storage/intent_interleaving_iter.go#L1275-L1277)
[pkg/storage/pebble_iterator.go#L610-L612](pkg/storage/pebble_iterator.go#L610-L612)
[external/com_github_cockroachdb_pebble/iterator.go#L1992-L1994](external/com_github_cockroachdb_pebble/iterator.go#L1992-L1994)
[external/com_github_cockroachdb_pebble/iterator.go#L2088-L2090](external/com_github_cockroachdb_pebble/iterator.go#L2088-L2090)
[external/com_github_cockroachdb_pebble/iterator.go#L1057-L1059](external/com_github_cockroachdb_pebble/iterator.go#L1057-L1059)
[external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L577-L579](external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L577-L579)
[external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L717-L719](external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L717-L719)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1290-L1292](external/com_github_cockroachdb_pebble/merging_iter.go#L1290-L1292)
[external/com_github_cockroachdb_pebble/merging_iter.go#L760-L762](external/com_github_cockroachdb_pebble/merging_iter.go#L760-L762)
[external/com_github_cockroachdb_pebble/level_iter.go#L804-L806](external/com_github_cockroachdb_pebble/level_iter.go#L804-L806)
[external/com_github_cockroachdb_pebble/level_iter.go#L879-L881](external/com_github_cockroachdb_pebble/level_iter.go#L879-L881)
[external/com_github_cockroachdb_pebble/level_iter.go#L567-L569](external/com_github_cockroachdb_pebble/level_iter.go#L567-L569)
[external/com_github_cockroachdb_pebble/file_cache.go#L565-L567](external/com_github_cockroachdb_pebble/file_cache.go#L565-L567)
[external/com_github_cockroachdb_pebble/file_cache.go#L685-L687](external/com_github_cockroachdb_pebble/file_cache.go#L685-L687)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L96-L98](external/com_github_cockroachdb_pebble/sstable/reader.go#L96-L98)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L133-L135](external/com_github_cockroachdb_pebble/sstable/reader.go#L133-L135)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L218-L220](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L218-L220)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L304-L306](external/com_github_cockroachdb_pebble/sstable/reader.go#L304-L306)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L312-L314](external/com_github_cockroachdb_pebble/sstable/reader.go#L312-L314)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511](external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413](external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413)
[external/com_github_cockroachdb_pebble/event.go#L1152-L1154](external/com_github_cockroachdb_pebble/event.go#L1152-L1154)
[external/com_github_cockroachdb_pebble/event.go#L1193-L1195](external/com_github_cockroachdb_pebble/event.go#L1193-L1195)
[external/com_github_cockroachdb_pebble/event.go#L993-L995](external/com_github_cockroachdb_pebble/event.go#L993-L995)
[pkg/storage/pebble.go#L1065-L1067](pkg/storage/pebble.go#L1065-L1067)

```
pkg/kv/kvserver/gc/gc.go in pkg/kv/kvserver/gc.processReplicatedKeyRange at line 453
pkg/kv/kvserver/rditer/replica_data_iter.go in pkg/kv/kvserver/rditer.IterateMVCCReplicaKeySpans at line 529
pkg/kv/kvserver/rditer/replica_data_iter.go in pkg/kv/kvserver/rditer.IterateMVCCReplicaKeySpans.func1 at line 526
pkg/kv/kvserver/gc/gc.go in pkg/kv/kvserver/gc.processReplicatedKeyRange.func1 at line 485
pkg/kv/kvserver/gc/gc_iterator.go in pkg/kv/kvserver/gc.(*gcIterator).state at line 93
pkg/kv/kvserver/gc/gc_iterator.go in pkg/kv/kvserver/gc.(*gcIterator).peekAt at line 120
pkg/kv/kvserver/gc/gc_iterator.go in pkg/kv/kvserver/gc.(*gcIterator).fillTo at line 160
pkg/storage/intent_interleaving_iter.go in pkg/storage.(*intentInterleavingIter).Prev at line 1276
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).Prev at line 611
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).Prev at line 1993
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).PrevWithLimit at line 2089
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).findPrevEntry at line 1058
external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go in github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).Prev at line 578
external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go in github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).prevPos at line 718
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).Prev at line 1291
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).prevEntry at line 761
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).Prev at line 805
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).skipEmptyFileBackward at line 880
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).loadFile at line 568
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).newIters at line 566
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).newPointIter at line 686
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).NewPointIter at line 97
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).newPointIter at line 134
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.newColumnBlockSingleLevelIterator at line 219
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readTopLevelIndexBlock at line 305
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readIndexBlock at line 313
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 510
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 412
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1153
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportFileCorruption at line 1194
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func2 at line 994
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1066
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.2 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.2 |
| Cockroach SHA | 0780d511bd527bfd9d02f4c7ca5ecc19115d9f5e |
| # of CPUs | 96 |
| # of Goroutines | 1501 |

Jira issue: CRDB-58040

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.