cockroachdb / cockroachdb/cockroach
Sentry: range_controller.go:977: unexpected next=30559420 > rewound_next_raft_index=30559419 (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:791 | [.....
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7610945091/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7610945091/?referrer=webhooks_plugin)
Panic Message:
```
range_controller.go:977: unexpected next=30559420 > rewound_next_raft_index=30559419
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:791
| [...repeated from below...]
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.constructRaftEventForReplica
| pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:977
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*rangeController).HandleRaftEventRaftMuLocked
| pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:1174
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/replica_rac2.(*processorImpl).HandleRaftReadyRaftMuLocked
| pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go:869
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| pkg/kv/kvserver/replica_raft.go:1039
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| pkg/kv/kvserver/replica_raft.go:875
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| pkg/kv/kvserver/store_raft.go:755
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| pkg/kv/kvserver/scheduler.go:419
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| pkg/kv/kvserver/scheduler.go:319
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:507
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (4) unexpected next=30559420 > rewound_next_raft_index=30559419
Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: unexpected next=30559420 > rewound_next_raft_index=30559419
range_controller.go:977: *withstack.withStack (top exception)
*assert.withAssertionFailure
panic.go:791: *withstack.withStack (1)
(check the extra data payloads)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/util/stop/stopper.go#L506-L508](pkg/util/stop/stopper.go#L506-L508)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1038-L1040](pkg/kv/kvserver/replica_raft.go#L1038-L1040)
[pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go#L868-L870](pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go#L868-L870)
[pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L1173-L1175](pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L1173-L1175)
[pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L976-L978](pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L976-L978)
[GOROOT/src/runtime/panic.go#L790-L792](GOROOT/src/runtime/panic.go#L790-L792)
[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/util/stop/stopper.go#L506-L508](pkg/util/stop/stopper.go#L506-L508)
[pkg/kv/kvserver/scheduler.go#L318-L320](pkg/kv/kvserver/scheduler.go#L318-L320)
[pkg/kv/kvserver/scheduler.go#L418-L420](pkg/kv/kvserver/scheduler.go#L418-L420)
[pkg/kv/kvserver/store_raft.go#L754-L756](pkg/kv/kvserver/store_raft.go#L754-L756)
[pkg/kv/kvserver/replica_raft.go#L874-L876](pkg/kv/kvserver/replica_raft.go#L874-L876)
[pkg/kv/kvserver/replica_raft.go#L1038-L1040](pkg/kv/kvserver/replica_raft.go#L1038-L1040)
[pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go#L868-L870](pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go#L868-L870)
[pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L1173-L1175](pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L1173-L1175)
[pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L976-L978](pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go#L976-L978)
```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 507
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1039
pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go in pkg/kv/kvserver/kvflowcontrol/replica_rac2.(*processorImpl).HandleRaftReadyRaftMuLocked at line 869
pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go in pkg/kv/kvserver/kvflowcontrol/rac2.(*rangeController).HandleRaftEventRaftMuLocked at line 1174
pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go in pkg/kv/kvserver/kvflowcontrol/rac2.constructRaftEventForReplica at line 977
GOROOT/src/runtime/panic.go in runtime.gopanic at line 791
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 507
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 319
pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 419
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 755
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 875
pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1039
pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go in pkg/kv/kvserver/kvflowcontrol/replica_rac2.(*processorImpl).HandleRaftReadyRaftMuLocked at line 869
pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go in pkg/kv/kvserver/kvflowcontrol/rac2.(*rangeController).HandleRaftEventRaftMuLocked at line 1174
pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go in pkg/kv/kvserver/kvflowcontrol/rac2.constructRaftEventForReplica at line 977
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.20 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v25.2.20 |
| Cockroach SHA | 38afacf970e63c9387d47f7555c2bb02b4d5cb02 |
| # of CPUs | 96 |
| # of Goroutines | 2597 |
Jira issue: CRDB-65789
Contributor guide
Research direction
Start in pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go at constructRaftEventForReplica line 977, then trace HandleRaftEventRaftMuLocked and processor.go line 869 through the Raft-ready path in replica_raft.go. The issue names no reproduction or test; investigate the Sentry panic and off-by-one Raft index first. Done means identifying the cause and preventing this assertion crash with regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100