cockroachdb / cockroachdb/cockroach

Sentry: pebble.go:1066: log.Fatal: local corruption detected: pebble: file 745633: block 3143017/3200: × checksum mismatch f4accab0 != 381a7f62 (1) attached stack trace -- stack trace: | github.c...

Open
#159,373 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.2 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7106965142/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7106965142/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1066: log.Fatal: local corruption detected: pebble: file 745633: block 3143017/3200: × checksum mismatch f4accab0 != 381a7f62
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1066
| github.com/cockroachdb/pebble.TeeEventListener.func2
| external/com_github_cockroachdb_pebble/event.go:994
| github.com/cockroachdb/pebble.(*DB).reportFileCorruption
| external/com_github_cockroachdb_pebble/event.go:1194
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1153
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:412
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:510
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:329
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:498
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1356
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1252
| github.com/cockroachdb/pebble.(*levelIter).Next
| external/com_github_cockroachdb_pebble/level_iter.go:753
| github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| external/com_github_cockroachdb_pebble/merging_iter.go:541
| github.com/cockroachdb/pebble.(*mergingIter).Next
| external/com_github_cockroachdb_pebble/merging_iter.go:1189
| github.com/cockroachdb/pebble.(*lazyCombinedIter).Next
| external/com_github_cockroachdb_pebble/range_keys.go:663
| github.com/cockroachdb/pebble.(*Iterator).mergeNext
| external/com_github_cockroachdb_pebble/iterator.go:1189
| github.com/cockroachdb/pebble.(*Iterator).mergeForward
| external/com_github_cockroachdb_pebble/iterator.go:723
| github.com/cockroachdb/pebble.(*Iterator).findNextEntry
| external/com_github_cockroachdb_pebble/iterator.go:618
| github.com/cockroachdb/pebble.(*Iterator).nextWithLimit
| external/com_github_cockroachdb_pebble/iterator.go:1985
| github.com/cockroachdb/pebble.(*Iterator).Next
| external/com_github_cockroachdb_pebble/iterator.go:1701
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).NextEngineKey
| pkg/storage/pebble_iterator.go:445
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Send.func4
| pkg/kv/kvserver/kv_snapshot_strategy.go:477
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.IterateReplicaKeySpans.func1
| pkg/kv/kvserver/rditer/replica_data_iter.go:437
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.IterateReplicaKeySpans
| pkg/kv/kvserver/rditer/replica_data_iter.go:440
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Send
| pkg/kv/kvserver/kv_snapshot_strategy.go:521
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.sendSnapshot
| pkg/kv/kvserver/store_snapshot.go:884
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).SendSnapshot
| pkg/kv/kvserver/raft_transport.go:1219
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).followerSendSnapshot.func5
| pkg/kv/kvserver/replica_command.go:3401
| github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
| pkg/util/timeutil/timeout.go:28
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).followerSendSnapshot
| pkg/kv/kvserver/replica_command.go:3399
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleDelegatedSnapshot
| pkg/kv/kvserver/store_raft.go:252
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).InternalDelegateRaftSnapshot
| pkg/kv/kvserver/raft_transport.go:616
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).DelegateRaftSnapshot
| pkg/kv/kvserver/raft_transport.go:579
Wraps: (2) secondary error attachment
| pebble: file 745633: block 3143017/3200: × checksum mismatch f4accab0 != 381a7f62
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:572
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:507
| | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:329
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:498
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1356
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1252
| | github.com/cockroachdb/pebble.(*levelIter).Next
| | external/com_github_cockroachdb_pebble/level_iter.go:753
| | github.com/cockroachdb/pebble.(*mergingIter).nextEntry
| | external/com_github_cockroachdb_pebble/merging_iter.go:541
| | github.com/cockroachdb/pebble.(*mergingIter).Next
| | external/com_github_cockroachdb_pebble/merging_iter.go:1189
| | github.com/cockroachdb/pebble.(*lazyCombinedIter).Next
| | external/com_github_cockroachdb_pebble/range_keys.go:663
| | github.com/cockroachdb/pebble.(*Iterator).mergeNext
| | external/com_github_cockroachdb_pebble/iterator.go:1189
| | github.com/cockroachdb/pebble.(*Iterator).mergeForward
| | external/com_github_cockroachdb_pebble/iterator.go:723
| | github.com/cockroachdb/pebble.(*Iterator).findNextEntry
| | external/com_github_cockroachdb_pebble/iterator.go:618
| | github.com/cockroachdb/pebble.(*Iterator).nextWithLimit
| | external/com_github_cockroachdb_pebble/iterator.go:1985
| | github.com/cockroachdb/pebble.(*Iterator).Next
| | external/com_github_cockroachdb_pebble/iterator.go:1701
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).NextEngineKey
| | pkg/storage/pebble_iterator.go:445
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Send.func4
| | pkg/kv/kvserver/kv_snapshot_strategy.go:477
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.IterateReplicaKeySpans.func1
| | pkg/kv/kvserver/rditer/replica_data_iter.go:437
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.IterateReplicaKeySpans
| | pkg/kv/kvserver/rditer/replica_data_iter.go:440
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Send
| | pkg/kv/kvserver/kv_snapshot_strategy.go:521
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.sendSnapshot
| | pkg/kv/kvserver/store_snapshot.go:884
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).SendSnapshot
| | pkg/kv/kvserver/raft_transport.go:1219
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).followerSendSnapshot.func5
| | pkg/kv/kvserver/replica_command.go:3401
| | github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
| | pkg/util/timeutil/timeout.go:28
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).followerSendSnapshot
| | pkg/kv/kvserver/replica_command.go:3399
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleDelegatedSnapshot
| | pkg/kv/kvserver/store_raft.go:252
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).InternalDelegateRaftSnapshot
| | pkg/kv...
```

Stacktrace (expand for inline code snippets):

[pkg/kv/kvserver/raft_transport.go#L578-L580](pkg/kv/kvserver/raft_transport.go#L578-L580)
[pkg/kv/kvserver/raft_transport.go#L615-L617](pkg/kv/kvserver/raft_transport.go#L615-L617)
[pkg/kv/kvserver/store_raft.go#L251-L253](pkg/kv/kvserver/store_raft.go#L251-L253)
[pkg/kv/kvserver/replica_command.go#L3398-L3400](pkg/kv/kvserver/replica_command.go#L3398-L3400)
[pkg/util/timeutil/timeout.go#L27-L29](pkg/util/timeutil/timeout.go#L27-L29)
[pkg/kv/kvserver/replica_command.go#L3400-L3402](pkg/kv/kvserver/replica_command.go#L3400-L3402)
[pkg/kv/kvserver/raft_transport.go#L1218-L1220](pkg/kv/kvserver/raft_transport.go#L1218-L1220)
[pkg/kv/kvserver/store_snapshot.go#L883-L885](pkg/kv/kvserver/store_snapshot.go#L883-L885)
[pkg/kv/kvserver/kv_snapshot_strategy.go#L520-L522](pkg/kv/kvserver/kv_snapshot_strategy.go#L520-L522)
[pkg/kv/kvserver/rditer/replica_data_iter.go#L439-L441](pkg/kv/kvserver/rditer/replica_data_iter.go#L439-L441)
[pkg/kv/kvserver/rditer/replica_data_iter.go#L436-L438](pkg/kv/kvserver/rditer/replica_data_iter.go#L436-L438)
[pkg/kv/kvserver/kv_snapshot_strategy.go#L476-L478](pkg/kv/kvserver/kv_snapshot_strategy.go#L476-L478)
[pkg/storage/pebble_iterator.go#L444-L446](pkg/storage/pebble_iterator.go#L444-L446)
[external/com_github_cockroachdb_pebble/iterator.go#L1700-L1702](external/com_github_cockroachdb_pebble/iterator.go#L1700-L1702)
[external/com_github_cockroachdb_pebble/iterator.go#L1984-L1986](external/com_github_cockroachdb_pebble/iterator.go#L1984-L1986)
[external/com_github_cockroachdb_pebble/iterator.go#L617-L619](external/com_github_cockroachdb_pebble/iterator.go#L617-L619)
[external/com_github_cockroachdb_pebble/iterator.go#L722-L724](external/com_github_cockroachdb_pebble/iterator.go#L722-L724)
[external/com_github_cockroachdb_pebble/iterator.go#L1188-L1190](external/com_github_cockroachdb_pebble/iterator.go#L1188-L1190)
[external/com_github_cockroachdb_pebble/range_keys.go#L662-L664](external/com_github_cockroachdb_pebble/range_keys.go#L662-L664)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1188-L1190](external/com_github_cockroachdb_pebble/merging_iter.go#L1188-L1190)
[external/com_github_cockroachdb_pebble/merging_iter.go#L540-L542](external/com_github_cockroachdb_pebble/merging_iter.go#L540-L542)
[external/com_github_cockroachdb_pebble/level_iter.go#L752-L754](external/com_github_cockroachdb_pebble/level_iter.go#L752-L754)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1251-L1253](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1251-L1253)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1355-L1357](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L1355-L1357)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L497-L499](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L497-L499)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L328-L330](external/com_github_cockroachdb_pebble/sstable/reader.go#L328-L330)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511](external/com_github_cockroachdb_pebble/sstable/block/block.go#L509-L511)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413](external/com_github_cockroachdb_pebble/sstable/block/block.go#L411-L413)
[external/com_github_cockroachdb_pebble/event.go#L1152-L1154](external/com_github_cockroachdb_pebble/event.go#L1152-L1154)
[external/com_github_cockroachdb_pebble/event.go#L1193-L1195](external/com_github_cockroachdb_pebble/event.go#L1193-L1195)
[external/com_github_cockroachdb_pebble/event.go#L993-L995](external/com_github_cockroachdb_pebble/event.go#L993-L995)
[pkg/storage/pebble.go#L1065-L1067](pkg/storage/pebble.go#L1065-L1067)

```
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).DelegateRaftSnapshot at line 579
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).InternalDelegateRaftSnapshot at line 616
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleDelegatedSnapshot at line 252
pkg/kv/kvserver/replica_command.go in pkg/kv/kvserver.(*Replica).followerSendSnapshot at line 3399
pkg/util/timeutil/timeout.go in pkg/util/timeutil.RunWithTimeout at line 28
pkg/kv/kvserver/replica_command.go in pkg/kv/kvserver.(*Replica).followerSendSnapshot.func5 at line 3401
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).SendSnapshot at line 1219
pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.sendSnapshot at line 884
pkg/kv/kvserver/kv_snapshot_strategy.go in pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Send at line 521
pkg/kv/kvserver/rditer/replica_data_iter.go in pkg/kv/kvserver/rditer.IterateReplicaKeySpans at line 440
pkg/kv/kvserver/rditer/replica_data_iter.go in pkg/kv/kvserver/rditer.IterateReplicaKeySpans.func1 at line 437
pkg/kv/kvserver/kv_snapshot_strategy.go in pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Send.func4 at line 477
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).NextEngineKey at line 445
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).Next at line 1701
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).nextWithLimit at line 1985
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).findNextEntry at line 618
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).mergeForward at line 723
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).mergeNext at line 1189
external/com_github_cockroachdb_pebble/range_keys.go in github.com/cockroachdb/pebble.(*lazyCombinedIter).Next at line 663
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).Next at line 1189
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).nextEntry at line 541
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).Next at line 753
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).Next at line 1252
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).skipForward at line 1356
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock at line 498
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock at line 329
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 510
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 412
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1153
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportFileCorruption at line 1194
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func2 at line 994
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1066
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.2 |
| Go Version | go1.23.7 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.2.2 |
| Cockroach SHA | 0780d511bd527bfd9d02f4c7ca5ecc19115d9f5e |
| # of CPUs | 8 |
| # of Goroutines | 1071 |

Jira issue: CRDB-57819

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.