cockroachdb / cockroachdb/cockroach

server: node catchup phase during startup

Open
#116,229 3 comments 0 reactions 0 assignees View on GitHub
C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Nodes that have been offline for extended periods of time may have a large backlog of work to do when rejoining the cluster. Currently, this work happens after the node is back online and receives leases, which can severely impact foreground traffic.

We should consider adding another server mode, e.g. `modeCatchup`, during startup which e.g.:

* Fails health checks.
* Prevents lease transfers and acquisition, and SQL connections.
* Catches up stale Raft replicas.
* Removes uninitialized and orphaned Raft replicas.
* Compacts down Pebble as needed.
* More?

It should also be possible to bypass this phase via e.g. environment variable, to get a node online asap.

https://github.com/cockroachdb/cockroach/blob/47f40bc27c9b7c76f3ed390601f33355eeb7c744/pkg/server/grpc_server.go#L51-L63

Jira issue: CRDB-34528

Contributor guide

Open the contributing guide

Research direction

Start with pkg/server/grpc_server.go at lines 51-63 and trace the startup path and existing server modes. Define the catchup phase scope, including health checks, lease behavior, SQL connections, Raft replica work, Pebble compaction, and the bypass environment variable. Done requires an agreed design and tests for the selected startup behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.