cockroachdb / cockroachdb/cockroach

Sentry: pebble.go:1066: log.Fatal: local corruption detected: pebble: file 000275: block 72195098/32354: × checksum mismatch 3f273106 != 1b987542 (1) attached stack trace -- stack trace: | github...

Open
#159,994 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.2 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7131819230/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7131819230/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1066: log.Fatal: local corruption detected: pebble: file 000275: block 72195098/32354: × checksum mismatch 3f273106 != 1b987542
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1066
| github.com/cockroachdb/pebble.TeeEventListener.func2
| external/com_github_cockroachdb_pebble/event.go:994
| github.com/cockroachdb/pebble.(*DB).reportFileCorruption
| external/com_github_cockroachdb_pebble/event.go:1194
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1153
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:412
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:510
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:329
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:498
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:757
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekPrefixGE
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:865
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekPrefixGE
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:808
| github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).SeekPrefixGE
| external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:323
| github.com/cockroachdb/pebble.(*levelIter).SeekPrefixGE
| external/com_github_cockroachdb_pebble/level_iter.go:671
| github.com/cockroachdb/pebble.(*mergingIter).seekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:969
| github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGEStrict
| external/com_github_cockroachdb_pebble/merging_iter.go:1044
| github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGE
| external/com_github_cockroachdb_pebble/merging_iter.go:1035
| github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekPrefixGE
| external/com_github_cockroachdb_pebble/range_keys.go:619
| github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE
| external/com_github_cockroachdb_pebble/iterator.go:1507
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| pkg/storage/pebble_iterator.go:343
| github.com/cockroachdb/cockroach/pkg/storage.mvccGetMetadata
| pkg/storage/mvcc.go:1686
| github.com/cockroachdb/cockroach/pkg/storage.mvccPutInternal
| pkg/storage/mvcc.go:2310
| github.com/cockroachdb/cockroach/pkg/storage.mvccPutUsingIter
| pkg/storage/mvcc.go:2062
| github.com/cockroachdb/cockroach/pkg/storage.mvccConditionalPutUsingIter
| pkg/storage/mvcc.go:3057
| github.com/cockroachdb/cockroach/pkg/storage.MVCCConditionalPut
| pkg/storage/mvcc.go:2964
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.ConditionalPut
| pkg/kv/kvserver/batcheval/cmd_conditional_put.go:92
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
| pkg/kv/kvserver/replica_evaluate.go:539
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| pkg/kv/kvserver/replica_evaluate.go:355
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
| pkg/kv/kvserver/replica_write.go:751
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
| pkg/kv/kvserver/replica_write.go:716
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch
| pkg/kv/kvserver/replica_write.go:479
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
| pkg/kv/kvserver/replica_proposal.go:999
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal
| pkg/kv/kvserver/replica_proposal.go:1095
Wraps: (2) secondary error attachment
| pebble: file 000275: block 72195098/32354: × checksum mismatch 3f273106 != 1b987542
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:572
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:507
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:329
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:498
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:757
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekPrefixGE
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:865
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:808
| | github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go:323
| | github.com/cockroachdb/pebble.(*levelIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/level_iter.go:671
| | github.com/cockroachdb/pebble.(*mergingIter).seekGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:969
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGEStrict
| | external/com_github_cockroachdb_pebble/merging_iter.go:1044
| | github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/merging_iter.go:1035
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/range_keys.go:619
| | github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE
| | external/com_github_cockroachdb_pebble/iterator.go:1507
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| | pkg/storage/pebble_iterator.go:343
| | github.com/cockroachdb/cockroach/pkg/storage.mvccGetMetadata
| | pkg/storage/mvcc.go:1686
| | github.com/cockroachdb/cockroach/pkg/storage.mvccPutInternal
| | pkg/storage/mvcc.go:2310
| | github.com/cockroachdb/cockroach/pkg/storage.mvccPutUsingIter
| | pkg/storage/mvcc.go:2062
| | github.com/cockroachdb/cockroach/pkg/storage.mvccConditionalPutUsingIter
| | pkg/storage/mvcc.go:3057
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCConditionalPut
| | pkg/storage/mvcc.go:2964
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.ConditionalPut
| | pkg/kv/kvserver/batcheval/cmd_conditional_put.go:92
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
| | pkg/kv/kvserver/replica_evaluate.go:539
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| | pkg/kv/kvserver/replica_evaluate.go:355
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper
| | pkg/kv/kvserver/replica_write.go:751
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes
| | pkg/kv/kvserver/replica_write.go:716
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateWriteBatch
| | pkg/kv/kvserver/replica_write.go:479
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal
| | pkg/kv/kvserver/replica_proposal.go:999
| | github.com/c...
```

Stacktrace (expand for inline code snippets):

[pkg/kv/kvserver/replica_proposal.go#L1094-L1096](pkg/kv/kvserver/replica_proposal.go#L1094-L1096)
[pkg/kv/kvserver/replica_proposal.go#L998-L1000](pkg/kv/kvserver/replica_proposal.go#L998-L1000)
[pkg/kv/kvserver/replica_write.go#L478-L480](pkg/kv/kvserver/replica_write.go#L478-L480)
[pkg/kv/kvserver/replica_write.go#L715-L717](pkg/kv/kvserver/replica_write.go#L715-L717)
[pkg/kv/kvserver/replica_write.go#L750-L752](pkg/kv/kvserver/replica_write.go#L750-L752)
[pkg/kv/kvserver/replica_evaluate.go#L354-L356](pkg/kv/kvserver/replica_evaluate.go#L354-L356)
[pkg/kv/kvserver/replica_evaluate.go#L538-L540](pkg/kv/kvserver/replica_evaluate.go#L538-L540)
[pkg/kv/kvserver/batcheval/cmd_conditional_put.go#L91-L93](pkg/kv/kvserver/batcheval/cmd_conditional_put.go#L91-L93)
[pkg/storage/mvcc.go#L2963-L2965](pkg/storage/mvcc.go#L2963-L2965)
[pkg/storage/mvcc.go#L3056-L3058](pkg/storage/mvcc.go#L3056-L3058)
[pkg/storage/mvcc.go#L2061-L2063](pkg/storage/mvcc.go#L2061-L2063)
[pkg/storage/mvcc.go#L2309-L2311](pkg/storage/mvcc.go#L2309-L2311)
[pkg/storage/mvcc.go#L1685-L1687](pkg/storage/mvcc.go#L1685-L1687)
[pkg/storage/pebble_iterator.go#L342-L344](pkg/storage/pebble_iterator.go#L342-L344)
[external/com_github_cockroachdb_pebble/iterator.go#L1506-L1508](external/com_github_cockroachdb_pebble/iterator.go#L1506-L1508)
[external/com_github_cockroachdb_pebble/range_keys.go#L618-L620](external/com_github_cockroachdb_pebble/range_keys.go#L618-L620)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1034-L1036](external/com_github_cockroachdb_pebble/merging_iter.go#L1034-L1036)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1043-L1045](external/com_github_cockroachdb_pebble/merging_iter.go#L1043-L1045)
[external/com_github_cockroachdb_pebble/merging_iter.go#L968-L970](external/com_github_cockroachdb_pebble/merging_iter.go#L968-L970)
[external/com_github_cockroachdb_pebble/level_iter.go#L670-L672](external/com_github_cockroachdb_pebble/level_iter.go#L670-L672)
[external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L322-L324](external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go#L322-L324)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L807-L809](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L807-L809)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L864-L866](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L864-L866)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L756-L758](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L756-L758)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L497-L499](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L497-L499)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L328-L330](external/com_github_cockroachdb_pebble/sstable/reader.go#L328-L330)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511](external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413](external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413)
[external/com_github_cockroachdb_pebble/event.go#L1152-L1154](external/com_github_cockroachdb_pebble/event.go#L1152-L1154)
[external/com_github_cockroachdb_pebble/event.go#L1193-L1195](external/com_github_cockroachdb_pebble/event.go#L1193-L1195)
[external/com_github_cockroachdb_pebble/event.go#L993-L995](external/com_github_cockroachdb_pebble/event.go#L993-L995)
[pkg/storage/pebble.go#L1065-L1067](pkg/storage/pebble.go#L1065-L1067)

```
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).requestToProposal at line 1095
pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).evaluateProposal at line 999
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatch at line 479
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWithServersideRefreshes at line 716
pkg/kv/kvserver/replica_write.go in pkg/kv/kvserver.(*Replica).evaluateWriteBatchWrapper at line 751
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 355
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 539
pkg/kv/kvserver/batcheval/cmd_conditional_put.go in pkg/kv/kvserver/batcheval.ConditionalPut at line 92
pkg/storage/mvcc.go in pkg/storage.MVCCConditionalPut at line 2964
pkg/storage/mvcc.go in pkg/storage.mvccConditionalPutUsingIter at line 3057
pkg/storage/mvcc.go in pkg/storage.mvccPutUsingIter at line 2062
pkg/storage/mvcc.go in pkg/storage.mvccPutInternal at line 2310
pkg/storage/mvcc.go in pkg/storage.mvccGetMetadata at line 1686
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).SeekGE at line 343
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekPrefixGE at line 1507
external/com_github_cockroachdb_pebble/range_keys.go in github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekPrefixGE at line 619
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGE at line 1035
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).SeekPrefixGEStrict at line 1044
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).seekGE at line 969
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).SeekPrefixGE at line 671
external/com_github_cockroachdb_pebble/internal/keyspan/interleaving_iter.go in github.com/cockroachdb/pebble/internal/keyspan.(*InterleavingIter).SeekPrefixGE at line 323
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekPrefixGE at line 808
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekPrefixGE at line 865
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper at line 757
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock at line 498
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock at line 329
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 510
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 412
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1153
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportFileCorruption at line 1194
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func2 at line 994
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1066
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.2 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.2 |
| Cockroach SHA | 0780d511bd527bfd9d02f4c7ca5ecc19115d9f5e |
| # of CPUs | 8 |
| # of Goroutines | 949 |

Jira issue: CRDB-58109

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.