cockroachdb / cockroachdb/cockroach
Sentry: pebble.go:1313: log.Dev.Fatal: local corruption detected: pebble: backing file 006333 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b (1) attached stack trace -- stack ...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7171929674/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7171929674/?referrer=webhooks_plugin)
Panic Message:
```
pebble.go:1313: log.Dev.Fatal: local corruption detected: pebble: backing file 006333 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1313
| github.com/cockroachdb/pebble.TeeEventListener.func6
| external/com_github_cockroachdb_pebble/event.go:1243
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1430
| github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
| external/com_github_cockroachdb_pebble/file_cache.go:285
| github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties-range1
| external/com_github_cockroachdb_pebble/table_stats.go:324
| github.com/cockroachdb/pebble/internal/manifest.(*btree[...]).All.func1
| external/com_github_cockroachdb_pebble/internal/manifest/btree.go:789
| github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties
| external/com_github_cockroachdb_pebble/table_stats.go:313
| github.com/cockroachdb/pebble.(*DB).collectTableStats
| external/com_github_cockroachdb_pebble/table_stats.go:124
| runtime.goexit
| src/runtime/asm_amd64.s:1700
Wraps: (2) secondary error attachment
| pebble: backing file 006333 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:489
| | [...repeated from below...]
| Wraps: (3) pebble: backing file 006333 error
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/blob.(*fileFooter).decode
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:435
| | github.com/cockroachdb/pebble/sstable/blob.NewFileReader
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:536
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:239
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:487
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:193
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:92
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
| | external/com_github_cockroachdb_pebble/file_cache.go:283
| | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties-range1
| | external/com_github_cockroachdb_pebble/table_stats.go:324
| | github.com/cockroachdb/pebble/internal/manifest.(*btree[...]).All.func1
| | external/com_github_cockroachdb_pebble/internal/manifest/btree.go:789
| | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties
| | external/com_github_cockroachdb_pebble/table_stats.go:313
| | github.com/cockroachdb/pebble.(*DB).collectTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:124
| | runtime.goexit
| | src/runtime/asm_amd64.s:1700
| Wraps: (6) invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
| Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.leafError
Wraps: (3) log.Dev.Fatal: local corruption detected: pebble: backing file 006333 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:489
| | [...repeated from below...]
| Wraps: (3) pebble: backing file 006333 error
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/blob.(*fileFooter).decode
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:435
| | github.com/cockroachdb/pebble/sstable/blob.NewFileReader
| | external/com_github_cockroachdb_pebble/sstable/blob/blob.go:536
| | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
| | external/com_github_cockroachdb_pebble/file_cache.go:239
| | github.com/cockroachdb/pebble.NewFileCache.func1
| | external/com_github_cockroachdb_pebble/file_cache.go:487
| | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
| | external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:193
| | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
| | external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:92
| | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
| | external/com_github_cockroachdb_pebble/file_cache.go:283
| | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties-range1
| | external/com_github_cockroachdb_pebble/table_stats.go:324
| | github.com/cockroachdb/pebble/internal/manifest.(*btree[...]).All.func1
| | external/com_github_cockroachdb_pebble/internal/manifest/btree.go:789
| | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties
| | external/com_github_cockroachdb_pebble/table_stats.go:313
| | github.com/cockroachdb/pebble.(*DB).collectTableStats
| | external/com_github_cockroachdb_pebble/table_stats.go:124
| | runtime.goexit
| | src/runtime/asm_amd64.s:1700
| Wraps: (6) invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
| Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.leafError
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: local corruption detected: pebble: backing file 006333 error: invalid blob file footer checksum 0x0000, expected: 0xd973cd2b
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
pebble.go:1313: *withstack.withStack (top exception)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_amd64.s#L1699-L1701](src/runtime/asm_amd64.s#L1699-L1701)
[external/com_github_cockroachdb_pebble/table_stats.go#L123-L125](external/com_github_cockroachdb_pebble/table_stats.go#L123-L125)
[external/com_github_cockroachdb_pebble/table_stats.go#L312-L314](external/com_github_cockroachdb_pebble/table_stats.go#L312-L314)
[external/com_github_cockroachdb_pebble/internal/manifest/btree.go#L788-L790](external/com_github_cockroachdb_pebble/internal/manifest/btree.go#L788-L790)
[external/com_github_cockroachdb_pebble/table_stats.go#L323-L325](external/com_github_cockroachdb_pebble/table_stats.go#L323-L325)
[external/com_github_cockroachdb_pebble/file_cache.go#L284-L286](external/com_github_cockroachdb_pebble/file_cache.go#L284-L286)
[external/com_github_cockroachdb_pebble/event.go#L1429-L1431](external/com_github_cockroachdb_pebble/event.go#L1429-L1431)
[external/com_github_cockroachdb_pebble/event.go#L1242-L1244](external/com_github_cockroachdb_pebble/event.go#L1242-L1244)
[pkg/storage/pebble.go#L1312-L1314](pkg/storage/pebble.go#L1312-L1314)
```
src/runtime/asm_amd64.s in runtime.goexit at line 1700
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).collectTableStats at line 124
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties at line 313
external/com_github_cockroachdb_pebble/internal/manifest/btree.go in github.com/cockroachdb/pebble/internal/manifest.(*btree[...]).All.func1 at line 789
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties-range1 at line 324
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob at line 285
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1430
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1243
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1313
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.4.2 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.4.2 |
| Cockroach SHA | 2c80e22e93a1fc4e9c268fa7b6c2018acfa09724 |
| # of CPUs | 20 |
| # of Goroutines | 478 |
Jira issue: CRDB-58452
Contributor guide
Assessment
This issue has not been assessed yet.