cockroachdb / cockroachdb/cockroach

Sentry: pebble.go:1313: log.Dev.Fatal: local corruption detected: pebble: file 013085: block 3441695/18146: × checksum mismatch 56b1e2c5 != 2c29c804 (1) attached stack trace -- stack trace: | git...

Open
#160,059 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.4 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7136406573/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7136406573/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1313: log.Dev.Fatal: local corruption detected: pebble: file 013085: block 3441695/18146: × checksum mismatch 56b1e2c5 != 2c29c804
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1313
| github.com/cockroachdb/pebble.TeeEventListener.func6
| external/com_github_cockroachdb_pebble/event.go:1243
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1430
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:318
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:452
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:394
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:507
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).NextPrefix
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1443
| github.com/cockroachdb/pebble.(*levelIter).NextPrefix
| external/com_github_cockroachdb_pebble/level_iter.go:774
| github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| external/com_github_cockroachdb_pebble/merging_iter.go:543
| github.com/cockroachdb/pebble.(*mergingIter).NextPrefix
| external/com_github_cockroachdb_pebble/merging_iter.go:1227
| github.com/cockroachdb/pebble.(*lazyCombinedIter).NextPrefix
| external/com_github_cockroachdb_pebble/range_keys.go:673
| github.com/cockroachdb/pebble.(*Iterator).internalNextPrefix
| external/com_github_cockroachdb_pebble/iterator.go:1910
| github.com/cockroachdb/pebble.(*Iterator).nextPrefix
| external/com_github_cockroachdb_pebble/iterator.go:1802
| github.com/cockroachdb/pebble.(*Iterator).NextPrefix
| external/com_github_cockroachdb_pebble/iterator.go:1768
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).NextKey
| pkg/storage/pebble_iterator.go:485
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).nextKey
| pkg/storage/pebble_mvcc_scanner.go:1112
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).advanceKeyForward
| pkg/storage/pebble_mvcc_scanner.go:1176
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).advance
| pkg/storage/pebble_mvcc_scanner.go:649
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).scan
| pkg/storage/pebble_mvcc_scanner.go:687
| github.com/cockroachdb/cockroach/pkg/storage.mvccScanToBytes
| pkg/storage/mvcc.go:4509
| github.com/cockroachdb/cockroach/pkg/storage.MVCCScanToBytes
| pkg/storage/mvcc.go:4866
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.Scan
| pkg/kv/kvserver/batcheval/cmd_scan.go:71
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
| pkg/kv/kvserver/replica_evaluate.go:543
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| pkg/kv/kvserver/replica_evaluate.go:356
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes
| pkg/kv/kvserver/replica_read.go:480
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
| pkg/kv/kvserver/replica_read.go:131
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:512
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:185
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:199
| github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| pkg/server/node.go:1667
Wraps: (2) secondary error attachment
| pebble: file 013085: block 3441695/18146: × checksum mismatch 56b1e2c5 != 2c29c804
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:523
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:449
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:394
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:507
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).NextPrefix
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1443
| | github.com/cockroachdb/pebble.(*levelIter).NextPrefix
| | external/com_github_cockroachdb_pebble/level_iter.go:774
| | github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| | external/com_github_cockroachdb_pebble/merging_iter.go:543
| | github.com/cockroachdb/pebble.(*mergingIter).NextPrefix
| | external/com_github_cockroachdb_pebble/merging_iter.go:1227
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).NextPrefix
| | external/com_github_cockroachdb_pebble/range_keys.go:673
| | github.com/cockroachdb/pebble.(*Iterator).internalNextPrefix
| | external/com_github_cockroachdb_pebble/iterator.go:1910
| | github.com/cockroachdb/pebble.(*Iterator).nextPrefix
| | external/com_github_cockroachdb_pebble/iterator.go:1802
| | github.com/cockroachdb/pebble.(*Iterator).NextPrefix
| | external/com_github_cockroachdb_pebble/iterator.go:1768
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).NextKey
| | pkg/storage/pebble_iterator.go:485
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).nextKey
| | pkg/storage/pebble_mvcc_scanner.go:1112
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).advanceKeyForward
| | pkg/storage/pebble_mvcc_scanner.go:1176
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).advance
| | pkg/storage/pebble_mvcc_scanner.go:649
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).scan
| | pkg/storage/pebble_mvcc_scanner.go:687
| | github.com/cockroachdb/cockroach/pkg/storage.mvccScanToBytes
| | pkg/storage/mvcc.go:4509
| | github.com/cockroachdb/cockroach/pkg/storage.MVCCScanToBytes
| | pkg/storage/mvcc.go:4866
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.Scan
| | pkg/kv/kvserver/batcheval/cmd_scan.go:71
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
| | pkg/kv/kvserver/replica_evaluate.go:543
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| | pkg/kv/kvserver/replica_evaluate.go:356
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes
| | pkg/kv/kvserver/replica_read.go:480
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
| | pkg/kv/kvserver/replica_read.go:131
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| | pkg/kv/kvserver/replica_send.go:512
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| | pkg/kv/kvserver/replica_send.go:185
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| | pkg/kv/kvserver/store_send.go:188
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| | pkg/kv/kvserver/stores.go:199
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
| | pkg/server/node.go:166...
```

Stacktrace (expand for inline code snippets):

[pkg/server/node.go#L1666-L1668](pkg/server/node.go#L1666-L1668)
[pkg/kv/kvserver/stores.go#L198-L200](pkg/kv/kvserver/stores.go#L198-L200)
[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L184-L186](pkg/kv/kvserver/replica_send.go#L184-L186)
[pkg/kv/kvserver/replica_send.go#L511-L513](pkg/kv/kvserver/replica_send.go#L511-L513)
[pkg/kv/kvserver/replica_read.go#L130-L132](pkg/kv/kvserver/replica_read.go#L130-L132)
[pkg/kv/kvserver/replica_read.go#L479-L481](pkg/kv/kvserver/replica_read.go#L479-L481)
[pkg/kv/kvserver/replica_evaluate.go#L355-L357](pkg/kv/kvserver/replica_evaluate.go#L355-L357)
[pkg/kv/kvserver/replica_evaluate.go#L542-L544](pkg/kv/kvserver/replica_evaluate.go#L542-L544)
[pkg/kv/kvserver/batcheval/cmd_scan.go#L70-L72](pkg/kv/kvserver/batcheval/cmd_scan.go#L70-L72)
[pkg/storage/mvcc.go#L4865-L4867](pkg/storage/mvcc.go#L4865-L4867)
[pkg/storage/mvcc.go#L4508-L4510](pkg/storage/mvcc.go#L4508-L4510)
[pkg/storage/pebble_mvcc_scanner.go#L686-L688](pkg/storage/pebble_mvcc_scanner.go#L686-L688)
[pkg/storage/pebble_mvcc_scanner.go#L648-L650](pkg/storage/pebble_mvcc_scanner.go#L648-L650)
[pkg/storage/pebble_mvcc_scanner.go#L1175-L1177](pkg/storage/pebble_mvcc_scanner.go#L1175-L1177)
[pkg/storage/pebble_mvcc_scanner.go#L1111-L1113](pkg/storage/pebble_mvcc_scanner.go#L1111-L1113)
[pkg/storage/pebble_iterator.go#L484-L486](pkg/storage/pebble_iterator.go#L484-L486)
[external/com_github_cockroachdb_pebble/iterator.go#L1767-L1769](external/com_github_cockroachdb_pebble/iterator.go#L1767-L1769)
[external/com_github_cockroachdb_pebble/iterator.go#L1801-L1803](external/com_github_cockroachdb_pebble/iterator.go#L1801-L1803)
[external/com_github_cockroachdb_pebble/iterator.go#L1909-L1911](external/com_github_cockroachdb_pebble/iterator.go#L1909-L1911)
[external/com_github_cockroachdb_pebble/range_keys.go#L672-L674](external/com_github_cockroachdb_pebble/range_keys.go#L672-L674)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1226-L1228](external/com_github_cockroachdb_pebble/merging_iter.go#L1226-L1228)
[external/com_github_cockroachdb_pebble/merging_iter.go#L542-L544](external/com_github_cockroachdb_pebble/merging_iter.go#L542-L544)
[external/com_github_cockroachdb_pebble/level_iter.go#L773-L775](external/com_github_cockroachdb_pebble/level_iter.go#L773-L775)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1442-L1444](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1442-L1444)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L506-L508](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L506-L508)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L393-L395](external/com_github_cockroachdb_pebble/sstable/reader.go#L393-L395)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L451-L453](external/com_github_cockroachdb_pebble/sstable/block/block.go#L451-L453)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L317-L319](external/com_github_cockroachdb_pebble/sstable/block/block.go#L317-L319)
[external/com_github_cockroachdb_pebble/event.go#L1429-L1431](external/com_github_cockroachdb_pebble/event.go#L1429-L1431)
[external/com_github_cockroachdb_pebble/event.go#L1242-L1244](external/com_github_cockroachdb_pebble/event.go#L1242-L1244)
[pkg/storage/pebble.go#L1312-L1314](pkg/storage/pebble.go#L1312-L1314)

```
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1667
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 199
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 512
pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatch at line 131
pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes at line 480
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 356
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 543
pkg/kv/kvserver/batcheval/cmd_scan.go in pkg/kv/kvserver/batcheval.Scan at line 71
pkg/storage/mvcc.go in pkg/storage.MVCCScanToBytes at line 4866
pkg/storage/mvcc.go in pkg/storage.mvccScanToBytes at line 4509
pkg/storage/pebble_mvcc_scanner.go in pkg/storage.(*pebbleMVCCScanner).scan at line 687
pkg/storage/pebble_mvcc_scanner.go in pkg/storage.(*pebbleMVCCScanner).advance at line 649
pkg/storage/pebble_mvcc_scanner.go in pkg/storage.(*pebbleMVCCScanner).advanceKeyForward at line 1176
pkg/storage/pebble_mvcc_scanner.go in pkg/storage.(*pebbleMVCCScanner).nextKey at line 1112
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).NextKey at line 485
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).NextPrefix at line 1768
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).nextPrefix at line 1802
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).internalNextPrefix at line 1910
external/com_github_cockroachdb_pebble/range_keys.go in github.com/cockroachdb/pebble.(*lazyCombinedIter).NextPrefix at line 673
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).NextPrefix at line 1227
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).nextEntry at line 543
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).NextPrefix at line 774
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).NextPrefix at line 1443
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock at line 507
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock at line 394
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 452
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 318
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1430
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1243
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1313
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.4.1 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.4.1 |
| Cockroach SHA | 60d802bd0bcb2c1aa81a4b5fd73fb4aa13bb5114 |
| # of CPUs | 32 |
| # of Goroutines | 707 |

Jira issue: CRDB-58131

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.