cockroachdb / cockroachdb/cockroach

Sentry: pebble.go:1436: log.Dev.Fatal: local corruption detected: pebble: file 000016: block 237440/12: × checksum mismatch fd0df90d != a1ac617 (1) attached stack trace -- stack trace: | github.c...

Open
#171,896 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-26.2 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7569470984/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7569470984/?referrer=webhooks_plugin)

Panic Message:

```
pebble.go:1436: log.Dev.Fatal: local corruption detected: pebble: file 000016: block 237440/12: × checksum mismatch fd0df90d != a1ac617
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
| pkg/storage/pebble.go:1436
| github.com/cockroachdb/pebble.TeeEventListener.func6
| external/com_github_cockroachdb_pebble/event.go:1243
| github.com/cockroachdb/pebble.(*DB).reportCorruption
| external/com_github_cockroachdb_pebble/event.go:1430
| github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption
| external/com_github_cockroachdb_pebble/sstable/block/block.go:331
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:426
| github.com/cockroachdb/pebble/sstable.(*Reader).readValueBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:456
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).ReadValueBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:530
| github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).getValueInternal
| external/com_github_cockroachdb_pebble/sstable/valblk/reader.go:275
| github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).FetchHandle
| external/com_github_cockroachdb_pebble/sstable/valblk/reader.go:218
| github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value
| external/com_github_cockroachdb_pebble/internal/base/value.go:79
| github.com/cockroachdb/pebble/internal/base.(*InternalKV).Value
| external/com_github_cockroachdb_pebble/internal/base/internal.go:718
| github.com/cockroachdb/pebble/internal/compact.(*Iter).saveValue
| external/com_github_cockroachdb_pebble/internal/compact/iterator.go:1400
| github.com/cockroachdb/pebble/internal/compact.(*Iter).setNext
| external/com_github_cockroachdb_pebble/internal/compact/iterator.go:873
| github.com/cockroachdb/pebble/internal/compact.(*Iter).Next
| external/com_github_cockroachdb_pebble/internal/compact/iterator.go:633
| github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable
| external/com_github_cockroachdb_pebble/internal/compact/run.go:264
| github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable
| external/com_github_cockroachdb_pebble/internal/compact/run.go:201
| github.com/cockroachdb/pebble.(*DB).compactAndWrite
| external/com_github_cockroachdb_pebble/compaction.go:2902
| github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction
| external/com_github_cockroachdb_pebble/compaction.go:2723
| github.com/cockroachdb/pebble.(*DB).runCompaction
| external/com_github_cockroachdb_pebble/compaction.go:2705
| github.com/cockroachdb/pebble.(*DB).compact1
| external/com_github_cockroachdb_pebble/compaction.go:2387
| github.com/cockroachdb/pebble.(*tableCompaction).Execute
| external/com_github_cockroachdb_pebble/compaction.go:408
| github.com/cockroachdb/pebble.(*DB).compact.func1.1
| external/com_github_cockroachdb_pebble/compaction.go:2255
| github.com/cockroachdb/pebble.(*DB).compact.func1
| external/com_github_cockroachdb_pebble/compaction.go:2284
| runtime/pprof.Do
| GOROOT/src/runtime/pprof/runtime.go:51
| github.com/cockroachdb/pebble.(*DB).compact
| external/com_github_cockroachdb_pebble/compaction.go:2249
| github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| external/com_github_cockroachdb_pebble/compaction.go:2118
| runtime.goexit
| src/runtime/asm_arm64.s:1268
Wraps: (2) secondary error attachment
| pebble: file 000016: block 237440/12: × checksum mismatch fd0df90d != a1ac617
| (1) ×
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:540
| | [...repeated from below...]
| Wraps: (3) pebble: file 000016
| Wraps: (4) forced error mark
| | ×
| | github.com/cockroachdb/errors/withstack/*withstack.withStack::
| Wraps: (5) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
| | external/com_github_cockroachdb_pebble/internal/base/error.go:35
| | github.com/cockroachdb/pebble/sstable/block.ValidateChecksum
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:193
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:538
| | github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| | external/com_github_cockroachdb_pebble/sstable/block/block.go:424
| | github.com/cockroachdb/pebble/sstable.(*Reader).readValueBlock
| | external/com_github_cockroachdb_pebble/sstable/reader.go:456
| | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).ReadValueBlock
| | external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:530
| | github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).getValueInternal
| | external/com_github_cockroachdb_pebble/sstable/valblk/reader.go:275
| | github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).FetchHandle
| | external/com_github_cockroachdb_pebble/sstable/valblk/reader.go:218
| | github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/lazy_value.go:195
| | github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value
| | external/com_github_cockroachdb_pebble/internal/base/value.go:79
| | github.com/cockroachdb/pebble/internal/base.(*InternalKV).Value
| | external/com_github_cockroachdb_pebble/internal/base/internal.go:718
| | github.com/cockroachdb/pebble/internal/compact.(*Iter).saveValue
| | external/com_github_cockroachdb_pebble/internal/compact/iterator.go:1400
| | github.com/cockroachdb/pebble/internal/compact.(*Iter).setNext
| | external/com_github_cockroachdb_pebble/internal/compact/iterator.go:873
| | github.com/cockroachdb/pebble/internal/compact.(*Iter).Next
| | external/com_github_cockroachdb_pebble/internal/compact/iterator.go:633
| | github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable
| | external/com_github_cockroachdb_pebble/internal/compact/run.go:264
| | github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable
| | external/com_github_cockroachdb_pebble/internal/compact/run.go:201
| | github.com/cockroachdb/pebble.(*DB).compactAndWrite
| | external/com_github_cockroachdb_pebble/compaction.go:2902
| | github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction
| | external/com_github_cockroachdb_pebble/compaction.go:2723
| | github.com/cockroachdb/pebble.(*DB).runCompaction
| | external/com_github_cockroachdb_pebble/compaction.go:2705
| | github.com/cockroachdb/pebble.(*DB).compact1
| | external/com_github_cockroachdb_pebble/compaction.go:2387
| | github.com/cockroachdb/pebble.(*tableCompaction).Execute
| | external/com_github_cockroachdb_pebble/compaction.go:408
| | github.com/cockroachdb/pebble.(*DB).compact.func1.1
| | external/com_github_cockroachdb_pebble/compaction.go:2255
| | github.com/cockroachdb/pebble.(*DB).compact.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2284
| | runtime/pprof.Do
| | GOROOT/src/runtime/pprof/runtime.go:51
| | github.com/cockroachdb/pebble.(*DB).compact
| | external/com_github_cockroachdb_pebble/compaction.go:2249
| | github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1
| | external/com_github_cockroachdb_pebble/compaction.go:2118
| | runtime.goexit
| | src/runtime/asm_arm64.s:1268
| Wraps: (6) block 237440/12: × checksum mismatch fd0df90d != a1ac617
| Error types: (1) *hintdetail.withHint (2...
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1267-L1269](src/runtime/asm_arm64.s#L1267-L1269)
[external/com_github_cockroachdb_pebble/compaction.go#L2117-L2119](external/com_github_cockroachdb_pebble/compaction.go#L2117-L2119)
[external/com_github_cockroachdb_pebble/compaction.go#L2248-L2250](external/com_github_cockroachdb_pebble/compaction.go#L2248-L2250)
[GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52)
[external/com_github_cockroachdb_pebble/compaction.go#L2283-L2285](external/com_github_cockroachdb_pebble/compaction.go#L2283-L2285)
[external/com_github_cockroachdb_pebble/compaction.go#L2254-L2256](external/com_github_cockroachdb_pebble/compaction.go#L2254-L2256)
[external/com_github_cockroachdb_pebble/compaction.go#L407-L409](external/com_github_cockroachdb_pebble/compaction.go#L407-L409)
[external/com_github_cockroachdb_pebble/compaction.go#L2386-L2388](external/com_github_cockroachdb_pebble/compaction.go#L2386-L2388)
[external/com_github_cockroachdb_pebble/compaction.go#L2704-L2706](external/com_github_cockroachdb_pebble/compaction.go#L2704-L2706)
[external/com_github_cockroachdb_pebble/compaction.go#L2722-L2724](external/com_github_cockroachdb_pebble/compaction.go#L2722-L2724)
[external/com_github_cockroachdb_pebble/compaction.go#L2901-L2903](external/com_github_cockroachdb_pebble/compaction.go#L2901-L2903)
[external/com_github_cockroachdb_pebble/internal/compact/run.go#L200-L202](external/com_github_cockroachdb_pebble/internal/compact/run.go#L200-L202)
[external/com_github_cockroachdb_pebble/internal/compact/run.go#L263-L265](external/com_github_cockroachdb_pebble/internal/compact/run.go#L263-L265)
[external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L632-L634](external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L632-L634)
[external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L872-L874](external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L872-L874)
[external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L1399-L1401](external/com_github_cockroachdb_pebble/internal/compact/iterator.go#L1399-L1401)
[external/com_github_cockroachdb_pebble/internal/base/internal.go#L717-L719](external/com_github_cockroachdb_pebble/internal/base/internal.go#L717-L719)
[external/com_github_cockroachdb_pebble/internal/base/value.go#L78-L80](external/com_github_cockroachdb_pebble/internal/base/value.go#L78-L80)
[external/com_github_cockroachdb_pebble/internal/base/lazy_value.go#L194-L196](external/com_github_cockroachdb_pebble/internal/base/lazy_value.go#L194-L196)
[external/com_github_cockroachdb_pebble/sstable/valblk/reader.go#L217-L219](external/com_github_cockroachdb_pebble/sstable/valblk/reader.go#L217-L219)
[external/com_github_cockroachdb_pebble/sstable/valblk/reader.go#L274-L276](external/com_github_cockroachdb_pebble/sstable/valblk/reader.go#L274-L276)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L529-L531](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L529-L531)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L455-L457](external/com_github_cockroachdb_pebble/sstable/reader.go#L455-L457)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L425-L427](external/com_github_cockroachdb_pebble/sstable/block/block.go#L425-L427)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L330-L332](external/com_github_cockroachdb_pebble/sstable/block/block.go#L330-L332)
[external/com_github_cockroachdb_pebble/event.go#L1429-L1431](external/com_github_cockroachdb_pebble/event.go#L1429-L1431)
[external/com_github_cockroachdb_pebble/event.go#L1242-L1244](external/com_github_cockroachdb_pebble/event.go#L1242-L1244)
[pkg/storage/pebble.go#L1435-L1437](pkg/storage/pebble.go#L1435-L1437)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1268
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runPickedCompaction.func1 at line 2118
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact at line 2249
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1 at line 2284
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1.1 at line 2255
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*tableCompaction).Execute at line 408
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact1 at line 2387
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runCompaction at line 2705
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).runDefaultTableCompaction at line 2723
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compactAndWrite at line 2902
external/com_github_cockroachdb_pebble/internal/compact/run.go in github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable at line 201
external/com_github_cockroachdb_pebble/internal/compact/run.go in github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable at line 264
external/com_github_cockroachdb_pebble/internal/compact/iterator.go in github.com/cockroachdb/pebble/internal/compact.(*Iter).Next at line 633
external/com_github_cockroachdb_pebble/internal/compact/iterator.go in github.com/cockroachdb/pebble/internal/compact.(*Iter).setNext at line 873
external/com_github_cockroachdb_pebble/internal/compact/iterator.go in github.com/cockroachdb/pebble/internal/compact.(*Iter).saveValue at line 1400
external/com_github_cockroachdb_pebble/internal/base/internal.go in github.com/cockroachdb/pebble/internal/base.(*InternalKV).Value at line 718
external/com_github_cockroachdb_pebble/internal/base/value.go in github.com/cockroachdb/pebble/internal/base.(*InternalValue).Value at line 79
external/com_github_cockroachdb_pebble/internal/base/lazy_value.go in github.com/cockroachdb/pebble/internal/base.(*LazyValue).Value at line 195
external/com_github_cockroachdb_pebble/sstable/valblk/reader.go in github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).FetchHandle at line 218
external/com_github_cockroachdb_pebble/sstable/valblk/reader.go in github.com/cockroachdb/pebble/sstable/valblk.(*valueBlockFetcher).getValueInternal at line 275
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).ReadValueBlock at line 530
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readValueBlock at line 456
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 426
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*ReadEnv).maybeReportCorruption at line 331
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1430
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1243
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1436
```

### Tags

| Tag | Value |
| --- | --- |
| Command | start-single-node |
| Environment | v26.2.0 |
| Go Version | go1.25.5 |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v26.2.0 |
| Cockroach SHA | 3d2773444f3ed07326a5daf43bc1555457d6964b |
| # of CPUs | 8 |
| # of Goroutines | 454 |

Jira issue: CRDB-65030

Contributor guide

Open the contributing guide

Research direction

Start with pkg/storage/pebble.go:1436 and the linked Sentry report, then trace the cited Pebble compaction and checksum-validation frames under external/com_github_cockroachdb_pebble. The report provides no reproduction steps or proposed change, so the cause and completion criteria must be established before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.