cockroachdb / cockroachdb/cockroach
Sentry: pebble.go:1313: log.Dev.Fatal: local corruption detected: pebble: backing file 009958 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b (1) attached stack trace -- stack ...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7168616138/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7168616138/?referrer=webhooks_plugin)
Panic Message:
```
pebble.go:1313: log.Dev.Fatal: local corruption detected: pebble: backing file 009958 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1313
| github.com/cockroachdb/pebble.TeeEventListener.func6
| external/com_github_cockroachdb_pebble/event.go:1243
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1430
| github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
| external/com_github_cockroachdb_pebble/file_cache.go:285
| github.com/cockroachdb/pebble.(*fileCacheHandle).GetValueReader
| external/com_github_cockroachdb_pebble/file_cache.go:399
| github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).retrieve
| external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:190
| github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).FetchHandle
| external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:126
| github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value
| external/com_github_cockroachdb_pebble/internal/base/value.go:79
| github.com/cockroachdb/pebble.(*Iterator).ValueAndErr
| external/com_github_cockroachdb_pebble/iterator.go:2308
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).UnsafeValue
| pkg/storage/pebble_iterator.go:526
| github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).UnsafeValue
| pkg/storage/intent_interleaving_iter.go:989
| github.com/cockroachdb/cockroach/pkg/storage.computeStatsForIterWithVisitors
| pkg/storage/mvcc.go:7390
| github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors
| pkg/storage/mvcc.go:7276
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors
| pkg/kv/kvserver/rditer/stats.go:90
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors
| pkg/kv/kvserver/rditer/stats.go:52
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.CalcReplicaDigest
| pkg/kv/kvserver/replica_consistency.go:624
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2
| pkg/kv/kvserver/replica_consistency.go:747
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1
| pkg/util/stop/stopper.go:440
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) secondary error attachment
| pebble: backing file 009958 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:489
| | [...repeated from below...]
| Wraps: (3) pebble: backing file 009958 error
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/blob.(*fileFooter).decode
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:435
| | github.com/cockroachdb/pebble/sstable/blob.NewFileReader
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:536
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:239
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:487
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:193
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:92
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
| | external/com_github_cockroachdb_pebble/file_cache.go:283
| | github.com/cockroachdb/pebble.(*fileCacheHandle).GetValueReader
| | external/com_github_cockroachdb_pebble/file_cache.go:399
| | github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).retrieve
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:190
| | github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).FetchHandle
| | external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go:126
| | github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| | github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/value.go:79
| | github.com/cockroachdb/pebble.(*Iterator).ValueAndErr
| | external/com_github_cockroachdb_pebble/iterator.go:2308
| | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).UnsafeValue
| | pkg/storage/pebble_iterator.go:526
| | github.com/cockroachdb/cockroach/pkg/storage.(*intentInterleavingIter).UnsafeValue
| | pkg/storage/intent_interleaving_iter.go:989
| | github.com/cockroachdb/cockroach/pkg/storage.computeStatsForIterWithVisitors
| | pkg/storage/mvcc.go:7390
| | github.com/cockroachdb/cockroach/pkg/storage.ComputeStatsWithVisitors
| | pkg/storage/mvcc.go:7276
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors
| | pkg/kv/kvserver/rditer/stats.go:90
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors
| | pkg/kv/kvserver/rditer/stats.go:52
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.CalcReplicaDigest
| | pkg/kv/kvserver/replica_consistency.go:624
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2
| | pkg/kv/kvserver/replica_consistency.go:747
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1
| | pkg/util/stop/stopper.go:440
| | runtime.goexit
| | src/runtime/asm_amd64.s:1700
| Wraps: (6) invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
| Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.leafError
Wraps: (3) log.Dev.Fatal: local corruption detected: pebble: backing file 009958 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:489
| | [...repeated from below...]
| Wraps: (3) pebble: backing file 009958 error
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/blob.(*fileFooter).decode
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:435
| | github.com/cockroachdb/pebble/sstable/blob.NewFileReader
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:536
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:239
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:487
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github...
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[pkg/util/stop/stopper.go#L439-L441](pkg/util/stop/stopper.go#L439-L441)
[pkg/kv/kvserver/replica_consistency.go#L746-L748](pkg/kv/kvserver/replica_consistency.go#L746-L748)
[pkg/kv/kvserver/replica_consistency.go#L623-L625](pkg/kv/kvserver/replica_consistency.go#L623-L625)
[pkg/kv/kvserver/rditer/stats.go#L51-L53](pkg/kv/kvserver/rditer/stats.go#L51-L53)
[pkg/kv/kvserver/rditer/stats.go#L89-L91](pkg/kv/kvserver/rditer/stats.go#L89-L91)
[pkg/storage/mvcc.go#L7275-L7277](pkg/storage/mvcc.go#L7275-L7277)
[pkg/storage/mvcc.go#L7389-L7391](pkg/storage/mvcc.go#L7389-L7391)
[pkg/storage/intent_interleaving_iter.go#L988-L990](pkg/storage/intent_interleaving_iter.go#L988-L990)
[pkg/storage/pebble_iterator.go#L525-L527](pkg/storage/pebble_iterator.go#L525-L527)
[external/com_github_cockroachdb_pebble/iterator.go#L2307-L2309](external/com_github_cockroachdb_pebble/iterator.go#L2307-L2309)
[external/com_github_cockroachdb_pebble/internal/base/value.go#L78-L80](external/com_github_cockroachdb_pebble/internal/base/value.go#L78-L80)
[external/com_github_cockroachdb_pebble/internal/base/lazy_value.go#L194-L196](external/com_github_cockroachdb_pebble/internal/base/lazy_value.go#L194-L196)
[external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go#L125-L127](external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go#L125-L127)
[external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go#L189-L191](external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go#L189-L191)
[external/com_github_cockroachdb_pebble/file_cache.go#L398-L400](external/com_github_cockroachdb_pebble/file_cache.go#L398-L400)
[external/com_github_cockroachdb_pebble/file_cache.go#L284-L286](external/com_github_cockroachdb_pebble/file_cache.go#L284-L286)
[external/com_github_cockroachdb_pebble/event.go#L1429-L1431](external/com_github_cockroachdb_pebble/event.go#L1429-L1431)
[external/com_github_cockroachdb_pebble/event.go#L1242-L1244](external/com_github_cockroachdb_pebble/event.go#L1242-L1244)
[pkg/storage/pebble.go#L1312-L1314](pkg/storage/pebble.go#L1312-L1314)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1 at line 440
pkg/kv/kvserver/replica_consistency.go in pkg/kv/kvserver.(*Replica).computeChecksumPostApply.func2 at line 747
pkg/kv/kvserver/replica_consistency.go in pkg/kv/kvserver.CalcReplicaDigest at line 624
pkg/kv/kvserver/rditer/stats.go in pkg/kv/kvserver/rditer.ComputeStatsForRangeWithVisitors at line 52
pkg/kv/kvserver/rditer/stats.go in pkg/kv/kvserver/rditer.computeStatsForSpansWithVisitors at line 90
pkg/storage/mvcc.go in pkg/storage.ComputeStatsWithVisitors at line 7276
pkg/storage/mvcc.go in pkg/storage.computeStatsForIterWithVisitors at line 7390
pkg/storage/intent_interleaving_iter.go in pkg/storage.(*intentInterleavingIter).UnsafeValue at line 989
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).UnsafeValue at line 526
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).ValueAndErr at line 2308
external/com_github_cockroachdb_pebble/internal/base/value.go in github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value at line 79
external/com_github_cockroachdb_pebble/internal/base/lazy_value.go in github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value at line 195
external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go in github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).FetchHandle at line 126
external/com_github_cockroachdb_pebble/sstable/blob/fetcher.go in github.com/cockroachdb/pebble/sstable/blob.(*ValueFetcher).retrieve at line 190
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).GetValueReader at line 399
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob at line 285
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1430
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1243
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1313
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.4.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.4.2 |
| Cockroach SHA | 2c80e22e93a1fc4e9c268fa7b6c2018acfa09724 |
| # of CPUs | 2 |
| # of Goroutines | 495 |
Jira issue: CRDB-58418
Contributor guide
Assessment
This issue has not been assessed yet.