cockroachdb / cockroachdb/cockroach

Sentry: panic.go:770: runtime error: slice bounds out of range [336:0] (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:770 | github.com/cockroachdb/co...

Open
#164,201 1 comment 0 reactions 0 assignees View on GitHub
A-storage branch-release-25.1 C-bug O-sentry T-storage X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7286835802/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7286835802/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:770: runtime error: slice bounds out of range [336:0]
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
| pkg/kv/kvserver/store_send.go:104
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| runtime.goPanicSliceB
| GOROOT/src/runtime/panic.go:154
| github.com/cockroachdb/pebble/sstable/block.Value.BlockData
| external/com_github_cockroachdb_pebble/sstable/block/buffer_pool.go:44
| github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead
| external/com_github_cockroachdb_pebble/sstable/block/block.go:507
| github.com/cockroachdb/pebble/sstable/block.(*Reader).Read
| external/com_github_cockroachdb_pebble/sstable/block/block.go:459
| github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader.go:336
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:528
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:787
| github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekGE
| external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:713
| github.com/cockroachdb/pebble.(*levelIter).SeekGE
| external/com_github_cockroachdb_pebble/level_iter.go:658
| github.com/cockroachdb/pebble.(*mergingIter).seekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:978
| github.com/cockroachdb/pebble.(*mergingIter).SeekGE
| external/com_github_cockroachdb_pebble/merging_iter.go:1026
| github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekGE
| external/com_github_cockroachdb_pebble/range_keys.go:606
| github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit
| external/com_github_cockroachdb_pebble/iterator.go:1348
| github.com/cockroachdb/pebble.(*Iterator).SeekGE
| external/com_github_cockroachdb_pebble/iterator.go:1252
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
| pkg/storage/pebble_iterator.go:351
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).iterSeek
| pkg/storage/pebble_mvcc_scanner.go:1722
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).seekToStartOfScan
| pkg/storage/pebble_mvcc_scanner.go:634
| github.com/cockroachdb/cockroach/pkg/storage.(*pebbleMVCCScanner).scan
| pkg/storage/pebble_mvcc_scanner.go:680
| github.com/cockroachdb/cockroach/pkg/storage.mvccScanToBytes
| pkg/storage/mvcc.go:4603
| github.com/cockroachdb/cockroach/pkg/storage.MVCCScanToBytes
| pkg/storage/mvcc.go:4959
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.Scan
| pkg/kv/kvserver/batcheval/cmd_scan.go:71
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand
| pkg/kv/kvserver/replica_evaluate.go:487
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch
| pkg/kv/kvserver/replica_evaluate.go:304
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes
| pkg/kv/kvserver/replica_read.go:471
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch
| pkg/kv/kvserver/replica_read.go:131
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
| pkg/kv/kvserver/replica_send.go:498
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
| pkg/kv/kvserver/replica_send.go:173
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
| pkg/kv/kvserver/store_send.go:188
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
| pkg/kv/kvserver/stores.go:199
Wraps: (2) runtime error: slice bounds out of range [336:0]
Error types: (1) *withstack.withStack (2) runtime.boundsError
-- report composition:
runtime.boundsError
panic.go:770: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[pkg/kv/kvserver/stores.go#L198-L200](pkg/kv/kvserver/stores.go#L198-L200)
[pkg/kv/kvserver/store_send.go#L187-L189](pkg/kv/kvserver/store_send.go#L187-L189)
[pkg/kv/kvserver/replica_send.go#L172-L174](pkg/kv/kvserver/replica_send.go#L172-L174)
[pkg/kv/kvserver/replica_send.go#L497-L499](pkg/kv/kvserver/replica_send.go#L497-L499)
[pkg/kv/kvserver/replica_read.go#L130-L132](pkg/kv/kvserver/replica_read.go#L130-L132)
[pkg/kv/kvserver/replica_read.go#L470-L472](pkg/kv/kvserver/replica_read.go#L470-L472)
[pkg/kv/kvserver/replica_evaluate.go#L303-L305](pkg/kv/kvserver/replica_evaluate.go#L303-L305)
[pkg/kv/kvserver/replica_evaluate.go#L486-L488](pkg/kv/kvserver/replica_evaluate.go#L486-L488)
[pkg/kv/kvserver/batcheval/cmd_scan.go#L70-L72](pkg/kv/kvserver/batcheval/cmd_scan.go#L70-L72)
[pkg/storage/mvcc.go#L4958-L4960](pkg/storage/mvcc.go#L4958-L4960)
[pkg/storage/mvcc.go#L4602-L4604](pkg/storage/mvcc.go#L4602-L4604)
[pkg/storage/pebble_mvcc_scanner.go#L679-L681](pkg/storage/pebble_mvcc_scanner.go#L679-L681)
[pkg/storage/pebble_mvcc_scanner.go#L633-L635](pkg/storage/pebble_mvcc_scanner.go#L633-L635)
[pkg/storage/pebble_mvcc_scanner.go#L1721-L1723](pkg/storage/pebble_mvcc_scanner.go#L1721-L1723)
[pkg/storage/pebble_iterator.go#L350-L352](pkg/storage/pebble_iterator.go#L350-L352)
[external/com_github_cockroachdb_pebble/iterator.go#L1251-L1253](external/com_github_cockroachdb_pebble/iterator.go#L1251-L1253)
[external/com_github_cockroachdb_pebble/iterator.go#L1347-L1349](external/com_github_cockroachdb_pebble/iterator.go#L1347-L1349)
[external/com_github_cockroachdb_pebble/range_keys.go#L605-L607](external/com_github_cockroachdb_pebble/range_keys.go#L605-L607)
[external/com_github_cockroachdb_pebble/merging_iter.go#L1025-L1027](external/com_github_cockroachdb_pebble/merging_iter.go#L1025-L1027)
[external/com_github_cockroachdb_pebble/merging_iter.go#L977-L979](external/com_github_cockroachdb_pebble/merging_iter.go#L977-L979)
[external/com_github_cockroachdb_pebble/level_iter.go#L657-L659](external/com_github_cockroachdb_pebble/level_iter.go#L657-L659)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L712-L714](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L712-L714)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L786-L788](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L786-L788)
[external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L527-L529](external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go#L527-L529)
[external/com_github_cockroachdb_pebble/sstable/reader.go#L335-L337](external/com_github_cockroachdb_pebble/sstable/reader.go#L335-L337)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L458-L460](external/com_github_cockroachdb_pebble/sstable/block/block.go#L458-L460)
[external/com_github_cockroachdb_pebble/sstable/block/block.go#L506-L508](external/com_github_cockroachdb_pebble/sstable/block/block.go#L506-L508)
[external/com_github_cockroachdb_pebble/sstable/block/buffer_pool.go#L43-L45](external/com_github_cockroachdb_pebble/sstable/block/buffer_pool.go#L43-L45)
[GOROOT/src/runtime/panic.go#L153-L155](GOROOT/src/runtime/panic.go#L153-L155)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)
[pkg/kv/kvserver/store_send.go#L103-L105](pkg/kv/kvserver/store_send.go#L103-L105)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)

```
pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 199
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 188
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 173
pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 498
pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatch at line 131
pkg/kv/kvserver/replica_read.go in pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes at line 471
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateBatch at line 304
pkg/kv/kvserver/replica_evaluate.go in pkg/kv/kvserver.evaluateCommand at line 487
pkg/kv/kvserver/batcheval/cmd_scan.go in pkg/kv/kvserver/batcheval.Scan at line 71
pkg/storage/mvcc.go in pkg/storage.MVCCScanToBytes at line 4959
pkg/storage/mvcc.go in pkg/storage.mvccScanToBytes at line 4603
pkg/storage/pebble_mvcc_scanner.go in pkg/storage.(*pebbleMVCCScanner).scan at line 680
pkg/storage/pebble_mvcc_scanner.go in pkg/storage.(*pebbleMVCCScanner).seekToStartOfScan at line 634
pkg/storage/pebble_mvcc_scanner.go in pkg/storage.(*pebbleMVCCScanner).iterSeek at line 1722
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).SeekGE at line 351
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGE at line 1252
external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit at line 1348
external/com_github_cockroachdb_pebble/range_keys.go in github.com/cockroachdb/pebble.(*lazyCombinedIter).SeekGE at line 606
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).SeekGE at line 1026
external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).seekGE at line 978
external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).SeekGE at line 658
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).SeekGE at line 713
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).seekGEHelper at line 787
external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock at line 528
external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock at line 336
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).Read at line 459
external/com_github_cockroachdb_pebble/sstable/block/block.go in github.com/cockroachdb/pebble/sstable/block.(*Reader).doRead at line 507
external/com_github_cockroachdb_pebble/sstable/block/buffer_pool.go in github.com/cockroachdb/pebble/sstable/block.Value.BlockData at line 44
GOROOT/src/runtime/panic.go in runtime.goPanicSliceB at line 154
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 104
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.1.5 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v25.1.5 |
| Cockroach SHA | d148d3abe6cb84ecc15c470c524f05c1e726c671 |
| # of CPUs | 8 |
| # of Goroutines | 540 |

Jira issue: CRDB-60683

Contributor guide

Open the contributing guide

Research direction

Start at external/com_github_cockroachdb_pebble/sstable/block/buffer_pool.go:44 and trace the reported call chain through pkg/storage/pebble_mvcc_scanner.go and pkg/kv/kvserver/batcheval/cmd_scan.go. Determine the cause of the slice-bounds panic in the v25.1.5 scan path; done means the crash is prevented and the relevant scan behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
22/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.