cockroachdb / cockroachdb/cockroach

Sentry: range_controller.go:977: unexpected next=30559420 > rewound_next_raft_index=30559419 (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | [.....

Open
#172,542 4 comments 0 reactions 0 assignees View on GitHub
A-kv branch-release-25.2 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7610945091/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7610945091/?referrer=webhooks_plugin)

Panic Message:

```
range_controller.go:977: unexpected next=30559420 > rewound_next_raft_index=30559419
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| [...repeated from below...]
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.constructRaftEventForReplica
| pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:977
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*rangeController).HandleRaftEventRaftMuLocked
| pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:1174
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/replica_rac2.(*processorImpl).HandleRaftReadyRaftMuLocked
| pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go:869
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1039
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:875
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:755
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:507
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (4) unexpected next=30559420 > rewound_next_raft_index=30559419
Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: unexpected next=30559420 > rewound_next_raft_index=30559419
range_controller.go:977: *withstack.withStack (top exception)
*assert.withAssertionFailure
panic.go:791: *withstack.withStack (1)
(check the extra data payloads)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/util/stop/stopper.go#L506-L508](pkg/util/stop/stopper.go#L506-L508)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1038-L1040](pkg/kv/kvserver/replica_raft.go#L1038-L1040)
[pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go#L868-L870](pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go#L868-L870)
[pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L1173-L1175](pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L1173-L1175)
[pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L976-L978](pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L976-L978)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/util/stop/stopper.go#L506-L508](pkg/util/stop/stopper.go#L506-L508)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1038-L1040](pkg/kv/kvserver/replica_raft.go#L1038-L1040)
[pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go#L868-L870](pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go#L868-L870)
[pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L1173-L1175](pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L1173-L1175)
[pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L976-L978](pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L976-L978)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 507
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1039
pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go in pkg/kv/kvserver/kvflowcontrol/replica_rac2.(*processorImpl).HandleRaftReadyRaftMuLocked at line 869
pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go in pkg/kv/kvserver/kvflowcontrol/rac2.(*rangeController).HandleRaftEventRaftMuLocked at line 1174
pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go in pkg/kv/kvserver/kvflowcontrol/rac2.constructRaftEventForReplica at line 977
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 507
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1039
pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go in pkg/kv/kvserver/kvflowcontrol/replica_rac2.(*processorImpl).HandleRaftReadyRaftMuLocked at line 869
pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go in pkg/kv/kvserver/kvflowcontrol/rac2.(*rangeController).HandleRaftEventRaftMuLocked at line 1174
pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go in pkg/kv/kvserver/kvflowcontrol/rac2.constructRaftEventForReplica at line 977
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.20 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v25.2.20 |
| Cockroach SHA | 38afacf970e63c9387d47f7555c2bb02b4d5cb02 |
| # of CPUs | 96 |
| # of Goroutines | 2597 |

Jira issue: CRDB-65789

Contributor guide

Open the contributing guide

Research direction

Start in pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go at constructRaftEventForReplica line 977, then trace HandleRaftEventRaftMuLocked and processor.go line 869 through the Raft-ready path in replica_raft.go. The issue names no reproduction or test; investigate the Sentry panic and off-by-one Raft index first. Done means identifying the cause and preventing this assertion crash with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.