cockroachdb / cockroachdb/cockroach
Sentry: stateloader.go:89: log.Dev.Fatal: unable to read Raft log entry 75 (×): pebble: file 000373: block 77726/10512: × checksum mismatch 9b0e7aba != 62194719 (1) attached stack trace -- stack t...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7609683635/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7609683635/?referrer=webhooks_plugin)
Panic Message:
```
stateloader.go:89: log.Dev.Fatal: unable to read Raft log entry 75 (×): pebble: file 000373: block 77726/10512: × checksum mismatch 9b0e7aba != 62194719
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.StateLoader.LoadLastEntryID
| pkg/kv/kvserver/logstore/stateloader.go:89
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvstorage.Replica.Load
| pkg/kv/kvserver/kvstorage/init.go:436
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start
| pkg/kv/kvserver/store.go:2329
| github.com/cockroachdb/cockroach/pkg/server.(*Node).start.func1
| pkg/server/node.go:693
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1
| pkg/util/stop/stopper.go:440
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) secondary error attachment
| pebble: file 000373: block 77726/10512: × checksum mismatch 9b0e7aba != 62194719
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:523
| | [...repeated from below...]
| Wraps: (2) pebble: file 000373
| Wraps: (3) . bit flip found: byte index 7314. got: 0x33. want: 0x32.
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:192
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:521
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:449
| | github.com/cockroachdb/pebble/sstable/blob.(*FileReader).ReadValueBlock
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:560
| | github.com/cockroachdb/pebble/sstable/blob.(*cachedReader).GetUnsafeValue
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:332
| | github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).retrieve
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:211
| | github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).FetchHandle
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:126
| | github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| | github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/value.go:79
| | github.com/cockroachdb/pebble.(*Iterator).ValueAndErr
| | external/com_github_cockroachdb_pebble/iterator.go:2308
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).UnsafeValue
| | pkg/storage/pebble_iterator.go:526
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.StateLoader.LoadLastEntryID
| | pkg/kv/kvserver/logstore/stateloader.go:87
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvstorage.Replica.Load
| | pkg/kv/kvserver/kvstorage/init.go:436
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start
| | pkg/kv/kvserver/store.go:2329
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).start.func1
| | pkg/server/node.go:693
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1
| | pkg/util/stop/stopper.go:440
| | runtime.goexit
| | src/runtime/asm_amd64.s:1700
| Wraps: (6) block 77726/10512: × checksum mismatch 9b0e7aba != 62194719
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *safedetails.withSafeDetails (4) *markers.withMark (5) *withstack.withStack (6) *errutil.leafError
Wraps: (3) log.Dev.Fatal: unable to read Raft log entry 75 (×): pebble: file 000373: block 77726/10512: × checksum mismatch 9b0e7aba != 62194719
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: unable to read Raft log entry 75 (×): pebble: file 000373: block 77726/10512: × checksum mismatch 9b0e7aba != 62194719
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
stateloader.go:89: *withstack.withStack (top exception)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L439-L441](pkg/util/stop/stopper.go#L439-L441)
[pkg/server/node.go#L692-L694](pkg/server/node.go#L692-L694)
[pkg/kv/kvserver/store.go#L2328-L2330](pkg/kv/kvserver/store.go#L2328-L2330)
[pkg/kv/kvserver/kvstorage/init.go#L435-L437](pkg/kv/kvserver/kvstorage/init.go#L435-L437)
[pkg/kv/kvserver/logstore/stateloader.go#L88-L90](pkg/kv/kvserver/logstore/stateloader.go#L88-L90)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1 at line 440
pkg/server/node.go in pkg/server.(*Node).start.func1 at line 693
pkg/kv/kvserver/store.go in pkg/kv/kvserver.(*Store).Start at line 2329
pkg/kv/kvserver/kvstorage/init.go in pkg/kv/kvserver/kvstorage.Replica.Load at line 436
pkg/kv/kvserver/logstore/stateloader.go in pkg/kv/kvserver/logstore.StateLoader.LoadLastEntryID at line 89
```
### Tags
| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v25.4.3 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.4.3 |
| Cockroach SHA | 71d853623f0d1f589fd5c727a1c4aec8a43e62e0 |
| # of CPUs | 4 |
| # of Goroutines | 141 |
Jira issue: CRDB-65752
Contributor guide
Research direction
Start with pkg/kv/kvserver/logstore/stateloader.go at lines 87-89, then trace the reported path through pkg/kv/kvserver/kvstorage/init.go and pkg/kv/kvserver/store.go. Investigate why the Pebble checksum mismatch occurs while loading Raft log entry 75 during startup. Done means the corruption scenario has a confirmed cause and a validated fix or clear handling behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100