cockroachdb / cockroachdb/cockroach

Sentry: pebble.go:1066: log.Fatal: local corruption detected: pebble: file 020756: block 19458196/32396: × checksum mismatch b0e4705 != 685d40e0 (1) attached stack trace -- stack trace: | github....

Open
#159,381 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.2 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7107191722/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7107191722/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1066: log.Fatal: local corruption detected: pebble: file 020756: block 19458196/32396: × checksum mismatch b0e4705 != 685d40e0
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1066
| github.com/cockroachdb/pebble.TeeEventListener.func2
| external/com_github_cockroachdb_pebble/event.go:994
| github.com/cockroachdb/pebble.(*DB).reportFileCorruption
| external/com_github_cockroachdb_pebble/event.go:1194
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1153
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:412
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:510
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:329
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:498
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:757
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekGE
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:683
| github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).SeekGE
| external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:287
| github.com/cockroachdb/pebble.(*levelIter).SeekGE
| external/com_github_cockroachdb_pebble/level_iter.go:645
| github.com/cockroachdb/pebble.(*mergingIter).seekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:978
| github.com/cockroachdb/pebble.(*mergingIter).SeekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:1026
| github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekGE
| external/com_github_cockroachdb_pebble/range_keys.go:606
| github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit
| external/com_github_cockroachdb_pebble/iterator.go:1350
| github.com/cockroachdb/pebble.(*Iterator).SeekGE
| external/com_github_cockroachdb_pebble/iterator.go:1254
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| pkg/storage/pebble_iterator.go:345
| github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).SeekGE
| pkg/storage/intent_interleaving_iter.go:547
| github.com/cockroachdb/cockroach/pkg/storage.(*MVCCIncrementalIterator).maybeSkipKeys
| pkg/storage/mvcc_incremental_iterator.go:445
| github.com/cockroachdb/cockroach/pkg/storage.(*MVCCIncrementalIterator).advance
| pkg/storage/mvcc_incremental_iterator.go:596
| github.com/cockroachdb/cockroach/pkg/storage.(*MVCCIncrementalIterator).Next
| pkg/storage/mvcc_incremental_iterator.go:348
| github.com/cockroachdb/cockroach/pkg/storage.mvccExportToWriter
| pkg/storage/mvcc.go:8222
| github.com/cockroachdb/cockroach/pkg/storage.MVCCExportToSST
| pkg/storage/mvcc.go:7816
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.evalExport
| pkg/kv/kvserver/batcheval/cmd_export.go:240
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
| pkg/kv/kvserver/replica_evaluate.go:541
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| pkg/kv/kvserver/replica_evaluate.go:355
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes
| pkg/kv/kvserver/replica_read.go:471
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
| pkg/kv/kvserver/replica_read.go:131
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:498
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:173
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
Wraps: (2) secondary error attachment
| pebble: file 020756: block 19458196/32396: × checksum mismatch b0e4705 != 685d40e0
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:572
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:507
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:329
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:498
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:757
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekGE
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:683
| | github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).SeekGE
| | external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:287
| | github.com/cockroachdb/pebble.(*levelIter).SeekGE
| | external/com_github_cockroachdb_pebble/level_iter.go:645
| | github.com/cockroachdb/pebble.(*mergingIter).seekGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:978
| | github.com/cockroachdb/pebble.(*mergingIter).SeekGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:1026
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekGE
| | external/com_github_cockroachdb_pebble/range_keys.go:606
| | github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit
| | external/com_github_cockroachdb_pebble/iterator.go:1350
| | github.com/cockroachdb/pebble.(*Iterator).SeekGE
| | external/com_github_cockroachdb_pebble/iterator.go:1254
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| | pkg/storage/pebble_iterator.go:345
| | github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).SeekGE
| | pkg/storage/intent_interleaving_iter.go:547
| | github.com/cockroachdb/cockroach/pkg/storage.(*MVCCIncrementalIterator).maybeSkipKeys
| | pkg/storage/mvcc_incremental_iterator.go:445
| | github.com/cockroachdb/cockroach/pkg/storage.(*MVCCIncrementalIterator).advance
| | pkg/storage/mvcc_incremental_iterator.go:596
| | github.com/cockroachdb/cockroach/pkg/storage.(*MVCCIncrementalIterator).Next
| | pkg/storage/mvcc_incremental_iterator.go:348
| | github.com/cockroachdb/cockroach/pkg/storage.mvccExportToWriter
| | pkg/storage/mvcc.go:8222
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCExportToSST
| | pkg/storage/mvcc.go:7816
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.evalExport
| | pkg/kv/kvserver/batcheval/cmd_export.go:240
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
| | pkg/kv/kvserver/replica_evaluate.go:541
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| | pkg/kv/kvserver/replica_evaluate.go:355
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes
| | pkg/kv/kvserver/replica_read.go:471
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
| | pkg/kv/kvserver/replica_read.go:131
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| | pkg/kv/kvserver/replica_send.go:498
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| | pkg/kv/kvserver/repl...
```

Stacktrace (expand for inline code snippets):

[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L172-L174](pkg/kv/kvserver/replica_send.go#L172-L174)
[pkg/kv/kvserver/replica_send.go#L497-L499](pkg/kv/kvserver/replica_send.go#L497-L499)
[pkg/kv/kvserver/replica_read.go#L130-L132](pkg/kv/kvserver/replica_read.go#L130-L132)
[pkg/kv/kvserver/replica_read.go#L470-L472](pkg/kv/kvserver/replica_read.go#L470-L472)
[pkg/kv/kvserver/replica_evaluate.go#L354-L356](pkg/kv/kvserver/replica_evaluate.go#L354-L356)
[pkg/kv/kvserver/replica_evaluate.go#L540-L542](pkg/kv/kvserver/replica_evaluate.go#L540-L542)
[pkg/kv/kvserver/batcheval/cmd_export.go#L239-L241](pkg/kv/kvserver/batcheval/cmd_export.go#L239-L241)
[pkg/storage/mvcc.go#L7815-L7817](pkg/storage/mvcc.go#L7815-L7817)
[pkg/storage/mvcc.go#L8221-L8223](pkg/storage/mvcc.go#L8221-L8223)
[pkg/storage/mvcc_incremental_iterator.go#L347-L349](pkg/storage/mvcc_incremental_iterator.go#L347-L349)
[pkg/storage/mvcc_incremental_iterator.go#L595-L597](pkg/storage/mvcc_incremental_iterator.go#L595-L597)
[pkg/storage/mvcc_incremental_iterator.go#L444-L446](pkg/storage/mvcc_incremental_iterator.go#L444-L446)
[pkg/storage/intent_interleaving_iter.go#L546-L548](pkg/storage/intent_interleaving_iter.go#L546-L548)
[pkg/storage/pebble_iterator.go#L344-L346](pkg/storage/pebble_iterator.go#L344-L346)
[external/com_github_cockroachdb_pebble/iterator.go#L1253-L1255](external/com_github_cockroachdb_pebble/iterator.go#L1253-L1255)
[external/com_github_cockroachdb_pebble/iterator.go#L1349-L1351](external/com_github_cockroachdb_pebble/iterator.go#L1349-L1351)
[external/com_github_cockroachdb_pebble/range_keys.go#L605-L607](external/com_github_cockroachdb_pebble/range_keys.go#L605-L607)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1025-L1027](external/com_github_cockroachdb_pebble/merging_iter.go#L1025-L1027)
[external/com_github_cockroachdb_pebble/merging_iter.go#L977-L979](external/com_github_cockroachdb_pebble/merging_iter.go#L977-L979)
[external/com_github_cockroachdb_pebble/level_iter.go#L644-L646](external/com_github_cockroachdb_pebble/level_iter.go#L644-L646)
[external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L286-L288](external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L286-L288)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L682-L684](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L682-L684)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L756-L758](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L756-L758)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L497-L499](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L497-L499)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L328-L330](external/com_github_cockroachdb_pebble/sstable/reader.go#L328-L330)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511](external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413](external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413)
[external/com_github_cockroachdb_pebble/event.go#L1152-L1154](external/com_github_cockroachdb_pebble/event.go#L1152-L1154)
[external/com_github_cockroachdb_pebble/event.go#L1193-L1195](external/com_github_cockroachdb_pebble/event.go#L1193-L1195)
[external/com_github_cockroachdb_pebble/event.go#L993-L995](external/com_github_cockroachdb_pebble/event.go#L993-L995)
[pkg/storage/pebble.go#L1065-L1067](pkg/storage/pebble.go#L1065-L1067)

```
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 173
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 498
pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatch at line 131
pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes at line 471
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 355
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 541
pkg/kv/kvserver/batcheval/cmd_export.go in pkg/kv/kvserver/batcheval.evalExport at line 240
pkg/storage/mvcc.go in pkg/storage.MVCCExportToSST at line 7816
pkg/storage/mvcc.go in pkg/storage.mvccExportToWriter at line 8222
pkg/storage/mvcc_incremental_iterator.go in pkg/storage.(*MVCCIncrementalIterator).Next at line 348
pkg/storage/mvcc_incremental_iterator.go in pkg/storage.(*MVCCIncrementalIterator).advance at line 596
pkg/storage/mvcc_incremental_iterator.go in pkg/storage.(*MVCCIncrementalIterator).maybeSkipKeys at line 445
pkg/storage/intent_interleaving_iter.go in pkg/storage.(*intentInterleavingIter).SeekGE at line 547
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).SeekGE at line 345
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGE at line 1254
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit at line 1350
external/com_github_cockroachdb_pebble/range_keys.go in github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekGE at line 606
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).SeekGE at line 1026
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).seekGE at line 978
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).SeekGE at line 645
external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go in github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).SeekGE at line 287
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekGE at line 683
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper at line 757
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock at line 498
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock at line 329
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 510
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 412
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1153
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportFileCorruption at line 1194
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func2 at line 994
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1066
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.2 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.2 |
| Cockroach SHA | 0780d511bd527bfd9d02f4c7ca5ecc19115d9f5e |
| # of CPUs | 8 |
| # of Goroutines | 957 |

Jira issue: CRDB-57822

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.