cockroachdb / cockroachdb/cockroach

server: start-up path is complex, masks problems

Open
#104,404 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement quality-friday T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

suggested by @knz.

**Is your feature request related to a problem? Please describe.**
The start-up sequence of a CockroachDB server is complicated. In https://github.com/cockroachdb/cockroach/pull/97710, we were dealing with the fact that operations during start-up must not return errors (or the process will exit) by adding appropriate retries.
Our protection against future code changes re-introducing some of the problems relies on randomized testing. This testing likely works fairly well, but only over longer time scales (i.e. possibly won't catch in the same PR).

I've also noticed that there are numerous synchronous writes to the KV DB in the path. If, for example, the `system.settings` table is unavailable, servers can't start. In that sense, loss of quorum on only that range is very close to loss of quorum on the entire cluster. The current architecture makes such shortcomings hard to spot.

**Describe the solution you'd like**

If there were a clearer separation between the critical start-up path and the rest of the code base, such problems could be prevented more easily. For example, rather than having a (few) large methods that carry out lots of operations in some more or less ad-hoc order, we could take a more structured approach that thins out the start-up path to the absolute essentials and carries out all other operations concurrently or in some fixed order, but without the ability to indefinitely block or terminate the start-up path.

Jira issue: CRDB-28511

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.