cockroachdb / cockroachdb/cockroach

server: when node takes significant time to start it gives misleading messages that it can't connect to cluster

Open
#109,518 2 comments 0 reactions 0 assignees View on GitHub
A-cli-server A-server-start-drain C-bug O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When node has lots of replicas and reloading state after shutdown takes significant time it will log:

```
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 The server appears to be unable to contact the other nodes in the cluster. Please try:
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +- starting the other nodes, if you haven't already;
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +- double-checking that the '--join' and '--listen'/'--advertise' flags are set up correctly;
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +- running the 'cockroach init' command if you are trying to initialize a new cluster.
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +
W230825 16:52:28.681762 10029 1@cli/start.go:567 ⋮ [n132] 30 +If problems persist, please see ‹https://www.cockroachlabs.com/docs/v22.2/cluster-setup-troubleshooting.html›.
```

which leads support/clients onto the goose chase for network issues.

What actually happening is that node successfully connects to the cluster and start loading replicas but the timeout triggers logging because server is not fully initialized yet.
At the same time, other nodes are already sending raft messages to this node if replicas were not removed from descriptors and receiving errors that node is not initialized.

**Environment:**
- CockroachDB version 22.2.9/23.1.x

Jira issue: CRDB-30961

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.