cockroachdb / cockroachdb/cockroach
server: when node takes significant time to start it gives misleading messages that it can't connect to cluster
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When node has lots of replicas and reloading state after shutdown takes significant time it will log:
```
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 The server appears to be unable to contact the other nodes in the cluster. Please try:
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +- starting the other nodes, if you haven't already;
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +- double-checking that the '--join' and '--listen'/'--advertise' flags are set up correctly;
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +- running the 'cockroach init' command if you are trying to initialize a new cluster.
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +If problems persist, please see ‹https://www.cockroachlabs.com/docs/v22.2/cluster-setup-troubleshooting.html›.
```
which leads support/clients onto the goose chase for network issues.
What actually happening is that node successfully connects to the cluster and start loading replicas but the timeout triggers logging because server is not fully initialized yet.
At the same time, other nodes are already sending raft messages to this node if replicas were not removed from descriptors and receiving errors that node is not initialized.
**Environment:**
- CockroachDB version 22.2.9/23.1.x
Jira issue: CRDB-30961
Contributor guide
Assessment
This issue has not been assessed yet.