cockroachdb / cockroachdb/cockroach
Sentry: node.go:765: log.Fatal: while initializing additional stores: error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve req...
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: [https://cockroach-labs.sentry.io/issues/7609390820/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7609390820/?referrer=webhooks_plugin)
Panic Message:
```
node.go:765: log.Fatal: while initializing additional stores: error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/server.(*Node).start.func3
| pkg/server/node.go:765
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:507
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (2) secondary error attachment
| error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).initializeAdditionalStores
| | pkg/server/node.go:991
| | [...repeated from below...]
| Wraps: (2) error allocating store ids
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/server.allocateStoreIDs
| | pkg/server/node.go:440
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).initializeAdditionalStores
| | pkg/server/node.go:989
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).start.func3
| | pkg/server/node.go:764
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | pkg/util/stop/stopper.go:507
| | runtime.goexit
| | src/runtime/asm_arm64.s:1223
| Wraps: (4) unable to allocate 2 store IDs for node 218
| Wraps: (5) replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
| Wraps: (6)
| | (opaque error wrapper)
| | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| | reportable 0:
| |
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaUnavailableError
| | pkg/kv/kvserver/replica_circuit_breaker.go:268
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).replicaUnavailableErrorRLocked
| | pkg/kv/kvserver/replica_circuit_breaker.go:289
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*leaderlessWatcher).refreshUnavailableState
| | pkg/kv/kvserver/replica.go:293
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
| | pkg/kv/kvserver/replica_raft.go:1514
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
| | pkg/kv/kvserver/store_raft.go:784
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:410
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:319
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | pkg/util/stop/stopper.go:507
| | runtime.goexit
| | src/runtime/asm_arm64.s:1223
| Wraps: (7) closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}
| Wraps: (8)
| | (opaque error wrapper)
| | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| | reportable 0:
| |
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*leaderlessWatcher).refreshUnavailableState
| | pkg/kv/kvserver/replica.go:294
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
| | pkg/kv/kvserver/replica_raft.go:1514
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
| | pkg/kv/kvserver/store_raft.go:784
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:410
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:319
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | pkg/util/stop/stopper.go:507
| | runtime.goexit
| | src/runtime/asm_arm64.s:1223
| Wraps: (9) replica has been leaderless for 1m0s
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *kvpb.ReplicaUnavailableError (6) *errbase.opaqueWrapper (7) *errutil.withPrefix (8) *errbase.opaqueWrapper (9) *errutil.leafError
Wraps: (3) log.Fatal: while initializing additional stores: error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: while initializing additional stores: error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
node.go:765: *withstack.withStack (top exception)
```
Stacktrace (expand for inline code snippets):
[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/util/stop/stopper.go#L506-L508](pkg/util/stop/stopper.go#L506-L508)
[pkg/server/node.go#L764-L766](pkg/server/node.go#L764-L766)
```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 507
pkg/server/node.go in pkg/server.(*Node).start.func3 at line 765
```
### Tags
| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.20 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v25.2.20 |
| Cockroach SHA | 38afacf970e63c9387d47f7555c2bb02b4d5cb02 |
| # of CPUs | 96 |
| # of Goroutines | 1344 |
Jira issue: CRDB-65747
Contributor guide
Research direction
Start with pkg/server/node.go at allocateStoreIDs (line 440), initializeAdditionalStores (lines 989-991), and Node.start (lines 764-765), then trace the replica-unavailable path in pkg/kv/kvserver/replica_circuit_breaker.go and replica.go cited by the stack trace. Use the Sentry report and Jira CRDB-65747 to establish the failure conditions; done means identifying the root cause and an agreed fix for the initialization failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100