cockroachdb / cockroachdb/cockroach
server: slow server init times can result in process manager timeouts
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
CockroachDB supports the [systemd readiness notification protocol](https://www.freedesktop.org/software/systemd/man/sd_notify.html).
We have observed that for clusters of a certain scale, initialization time can take a significant amount of time to complete (e.g. > 90s). Depending on how the process manager is configured, this can result in a situation where a node cannot restart and rejoin a cluster.
In cases where multiple nodes become unavailable at once, this can exacerbate unavailability and impeded recovery time.
A sanity check on the init process during startup, especially with large clusters, would be worthwhile here.
**Additional context**
Related to https://github.com/cockroachlabs/support/issues/2287.
Jira issue: CRDB-28416
Contributor guide
Assessment
This issue has not been assessed yet.