cockroachdb / cockroachdb/cockroach

Sentry: node.go:765: log.Fatal: while initializing additional stores: error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve req...

Open
#172,503 1 comment 0 reactions 0 assignees View on GitHub
branch-release-25.2 C-bug O-sentry T-kv X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: [https://cockroach-labs.sentry.io/issues/7609390820/?referrer=webhooks_plugin](https://cockroach-labs.sentry.io/issues/7609390820/?referrer=webhooks_plugin)

Panic Message:

```
node.go:765: log.Fatal: while initializing additional stores: error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/server.(*Node).start.func3
| pkg/server/node.go:765
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| pkg/util/stop/stopper.go:507
| runtime.goexit
| src/runtime/asm_arm64.s:1223
Wraps: (2) secondary error attachment
| error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).initializeAdditionalStores
| | pkg/server/node.go:991
| | [...repeated from below...]
| Wraps: (2) error allocating store ids
| Wraps: (3) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/server.allocateStoreIDs
| | pkg/server/node.go:440
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).initializeAdditionalStores
| | pkg/server/node.go:989
| | github.com/cockroachdb/cockroach/pkg/server.(*Node).start.func3
| | pkg/server/node.go:764
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | pkg/util/stop/stopper.go:507
| | runtime.goexit
| | src/runtime/asm_arm64.s:1223
| Wraps: (4) unable to allocate 2 store IDs for node 218
| Wraps: (5) replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
| Wraps: (6)
| | (opaque error wrapper)
| | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| | reportable 0:
| |
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaUnavailableError
| | pkg/kv/kvserver/replica_circuit_breaker.go:268
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).replicaUnavailableErrorRLocked
| | pkg/kv/kvserver/replica_circuit_breaker.go:289
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*leaderlessWatcher).refreshUnavailableState
| | pkg/kv/kvserver/replica.go:293
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
| | pkg/kv/kvserver/replica_raft.go:1514
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
| | pkg/kv/kvserver/store_raft.go:784
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:410
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:319
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | pkg/util/stop/stopper.go:507
| | runtime.goexit
| | src/runtime/asm_arm64.s:1223
| Wraps: (7) closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}
| Wraps: (8)
| | (opaque error wrapper)
| | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
| | reportable 0:
| |
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*leaderlessWatcher).refreshUnavailableState
| | pkg/kv/kvserver/replica.go:294
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).tick
| | pkg/kv/kvserver/replica_raft.go:1514
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processTick
| | pkg/kv/kvserver/store_raft.go:784
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | pkg/kv/kvserver/scheduler.go:410
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | pkg/kv/kvserver/scheduler.go:319
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | pkg/util/stop/stopper.go:507
| | runtime.goexit
| | src/runtime/asm_arm64.s:1223
| Wraps: (9) replica has been leaderless for 1m0s
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *kvpb.ReplicaUnavailableError (6) *errbase.opaqueWrapper (7) *errutil.withPrefix (8) *errbase.opaqueWrapper (9) *errutil.leafError
Wraps: (3) log.Fatal: while initializing additional stores: error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: while initializing additional stores: error allocating store ids: unable to allocate 2 store IDs for node 218: replica unavailable: (n212,s1670):238 unable to serve request to r3:× [(n212,s1670):238, (n209,s1664):239, (n211,s1672):240, (n210,s1667):241, (n207,s1659):242, next=243, gen=844]: closed timestamp: 1784005777.679645113,0 (2026-07-14 05:09:37); raft status: {"id":"ee","term":1251,"vote":"f2","commit":280585383,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":280585383,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
node.go:765: *withstack.withStack (top exception)
```

Stacktrace (expand for inline code snippets):

[src/runtime/asm_arm64.s#L1222-L1224](src/runtime/asm_arm64.s#L1222-L1224)
[pkg/util/stop/stopper.go#L506-L508](pkg/util/stop/stopper.go#L506-L508)
[pkg/server/node.go#L764-L766](pkg/server/node.go#L764-L766)

```
src/runtime/asm_arm64.s in runtime.goexit at line 1223
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 507
pkg/server/node.go in pkg/server.(*Node).start.func3 at line 765
```

### Tags

| Tag | Value |
| --- | --- |
| Command | server |
| Environment | v25.2.20 |
| Go Version | go1.23.12 X:nocoverageredesign |
| Platform | linux arm64 |
| Distribution | CCL |
| Cockroach Release | v25.2.20 |
| Cockroach SHA | 38afacf970e63c9387d47f7555c2bb02b4d5cb02 |
| # of CPUs | 96 |
| # of Goroutines | 1344 |

Jira issue: CRDB-65747

Contributor guide

Open the contributing guide

Research direction

Start with pkg/server/node.go at allocateStoreIDs (line 440), initializeAdditionalStores (lines 989-991), and Node.start (lines 764-765), then trace the replica-unavailable path in pkg/kv/kvserver/replica_circuit_breaker.go and replica.go cited by the stack trace. Use the Sentry report and Jira CRDB-65747 to establish the failure conditions; done means identifying the root cause and an agreed fix for the initialization failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.