cockroachdb / cockroachdb/cockroach

kvpb: v24.3.18: index out of bounds when marshaling RangeFeedCheckpoint

Open
#158,166 4 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed branch-release-24.3 C-bug O-sentry T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7049249525/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7049249525/?referrer=webhooks_plugin)

Panic Message:

```
panic.go:770: runtime error: index out of range [-1]
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:770
| runtime.goPanicIndex
| GOROOT/src/runtime/panic.go:114
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.encodeVarintApi
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:21580
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*RangeFeedCheckpoint).MarshalToSizedBuffer
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:20272
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*RangeFeedEvent).MarshalToSizedBuffer
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:20525
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*MuxRangeFeedEvent).MarshalToSizedBuffer
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:20581
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*MuxRangeFeedEvent).Marshal
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:20553
| github.com/cockroachdb/cockroach/pkg/rpc.codec.Marshal
| pkg/rpc/codec.go:29
| google.golang.org/grpc.encode
| external/org_golang_google_grpc/rpc_util.go:632
| google.golang.org/grpc.prepareMsg
| external/org_golang_google_grpc/stream.go:1766
| google.golang.org/grpc.(*serverStream).SendMsg
| external/org_golang_google_grpc/stream.go:1642
| github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*internalMuxRangeFeedServer).Send
| bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10855
| github.com/cockroachdb/cockroach/pkg/server.(*lockedMuxStream).Send
| pkg/server/node.go:2048
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*UnbufferedSender).SendUnbuffered
| pkg/kv/kvserver/rangefeed/unbuffered_sender.go:189
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*PerRangeEventSink).SendUnbuffered
| pkg/kv/kvserver/rangefeed/stream.go:64
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*bufferedRegistration).outputLoop
| pkg/kv/kvserver/rangefeed/buffered_registration.go:207
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*bufferedRegistration).runOutputLoop
| pkg/kv/kvserver/rangefeed/buffered_registration.go:231
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*ScheduledProcessor).Register.func1.1
| pkg/kv/kvserver/rangefeed/scheduled_processor.go:348
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:498
| runtime.goexit
| src/runtime/asm_amd64.s:1695
Wraps: (2) runtime error: index out of range [-1]
Error types: (1) *withstack.withStack (2) runtime.boundsError
-- report composition:
runtime.boundsError
panic.go:770: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696)
[pkg/util/stop/stopper.go#L497-L499](pkg/util/stop/stopper.go#L497-L499)
[pkg/kv/kvserver/rangefeed/scheduled_processor.go#L347-L349](pkg/kv/kvserver/rangefeed/scheduled_processor.go#L347-L349)
[pkg/kv/kvserver/rangefeed/buffered_registration.go#L230-L232](pkg/kv/kvserver/rangefeed/buffered_registration.go#L230-L232)
[pkg/kv/kvserver/rangefeed/buffered_registration.go#L206-L208](pkg/kv/kvserver/rangefeed/buffered_registration.go#L206-L208)
[pkg/kv/kvserver/rangefeed/stream.go#L63-L65](pkg/kv/kvserver/rangefeed/stream.go#L63-L65)
[pkg/kv/kvserver/rangefeed/unbuffered_sender.go#L188-L190](pkg/kv/kvserver/rangefeed/unbuffered_sender.go#L188-L190)
[pkg/server/node.go#L2047-L2049](pkg/server/node.go#L2047-L2049)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L10854-L10856](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L10854-L10856)
[external/org_golang_google_grpc/stream.go#L1641-L1643](external/org_golang_google_grpc/stream.go#L1641-L1643)
[external/org_golang_google_grpc/stream.go#L1765-L1767](external/org_golang_google_grpc/stream.go#L1765-L1767)
[external/org_golang_google_grpc/rpc_util.go#L631-L633](external/org_golang_google_grpc/rpc_util.go#L631-L633)
[pkg/rpc/codec.go#L28-L30](pkg/rpc/codec.go#L28-L30)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L20552-L20554](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L20552-L20554)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L20580-L20582](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L20580-L20582)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L20524-L20526](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L20524-L20526)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L20271-L20273](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L20271-L20273)
[bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L21579-L21581](bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/https://github.com/cockroachdb/cockroach/blob/37439c410d4583708c898b13a2f9e906f444867b/pkg/kv/kvpb/api.pb.go#L21579-L21581)
[GOROOT/src/runtime/panic.go#L113-L115](GOROOT/src/runtime/panic.go#L113-L115)
[GOROOT/src/runtime/panic.go#L769-L771](GOROOT/src/runtime/panic.go#L769-L771)

```
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 498
pkg/kv/kvserver/rangefeed/scheduled_processor.go in pkg/kv/kvserver/rangefeed.(*ScheduledProcessor).Register.func1.1 at line 348
pkg/kv/kvserver/rangefeed/buffered_registration.go in pkg/kv/kvserver/rangefeed.(*bufferedRegistration).runOutputLoop at line 231
pkg/kv/kvserver/rangefeed/buffered_registration.go in pkg/kv/kvserver/rangefeed.(*bufferedRegistration).outputLoop at line 207
pkg/kv/kvserver/rangefeed/stream.go in pkg/kv/kvserver/rangefeed.(*PerRangeEventSink).SendUnbuffered at line 64
pkg/kv/kvserver/rangefeed/unbuffered_sender.go in pkg/kv/kvserver/rangefeed.(*UnbufferedSender).SendUnbuffered at line 189
pkg/server/node.go in pkg/server.(*lockedMuxStream).Send at line 2048
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb.(*internalMuxRangeFeedServer).Send at line 10855
external/org_golang_google_grpc/stream.go in google.golang.org/grpc.(*serverStream).SendMsg at line 1642
external/org_golang_google_grpc/stream.go in google.golang.org/grpc.prepareMsg at line 1766
external/org_golang_google_grpc/rpc_util.go in google.golang.org/grpc.encode at line 632
pkg/rpc/codec.go in pkg/rpc.codec.Marshal at line 29
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb.(*MuxRangeFeedEvent).Marshal at line 20553
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb.(*MuxRangeFeedEvent).MarshalToSizedBuffer at line 20581
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb.(*RangeFeedEvent).MarshalToSizedBuffer at line 20525
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb.(*RangeFeedCheckpoint).MarshalToSizedBuffer at line 20272
bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go in pkg/kv/kvpb.encodeVarintApi at line 21580
GOROOT/src/runtime/panic.go in runtime.goPanicIndex at line 114
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v24.3.18 |
| Go Version | go1.22.8 X:nocoverageredesign |
| Platform | linux amd64 |
| Distribution | CCL |
| Cockroach Release | v24.3.18 |
| Cockroach SHA | 37439c410d4583708c898b13a2f9e906f444867b |
| # of CPUs | 8 |
| # of Goroutines | 1033 |

Jira issue: CRDB-57138

Contributor guide

Open the contributing guide

Research direction

Start with the RangeFeedCheckpoint.MarshalToSizedBuffer and encodeVarintApi frames in pkg/kv/kvpb/api.pb.go, then trace the event through pkg/kv/kvserver/rangefeed/unbuffered_sender.go. Reproduce the v24.3.18 crash if possible and determine what checkpoint data causes the bounds failure. Done means the reported RangeFeed event can be marshaled without an index-out-of-bounds panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.