cockroachdb / cockroachdb/cockroach

server: slow server init times can result in process manager timeouts

Open
#104,220 3 comments 0 reactions 0 assignees View on GitHub
A-server-start-drain C-investigation O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

CockroachDB supports the [systemd readiness notification protocol](https://www.freedesktop.org/software/systemd/man/sd_notify.html).

We have observed that for clusters of a certain scale, initialization time can take a significant amount of time to complete (e.g. > 90s). Depending on how the process manager is configured, this can result in a situation where a node cannot restart and rejoin a cluster.

In cases where multiple nodes become unavailable at once, this can exacerbate unavailability and impeded recovery time.

A sanity check on the init process during startup, especially with large clusters, would be worthwhile here.

**Additional context**

Related to https://github.com/cockroachlabs/support/issues/2287.

Jira issue: CRDB-28416

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.