cockroachdb / cockroachdb/cockroach
server: node catchup phase during startup
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Nodes that have been offline for extended periods of time may have a large backlog of work to do when rejoining the cluster. Currently, this work happens after the node is back online and receives leases, which can severely impact foreground traffic.
We should consider adding another server mode, e.g. `modeCatchup`, during startup which e.g.:
* Fails health checks.
* Prevents lease transfers and acquisition, and SQL connections.
* Catches up stale Raft replicas.
* Removes uninitialized and orphaned Raft replicas.
* Compacts down Pebble as needed.
* More?
It should also be possible to bypass this phase via e.g. environment variable, to get a node online asap.
https://github.com/cockroachdb/cockroach/blob/47f40bc27c9b7c76f3ed390601f33355eeb7c744/pkg/server/grpc_server.go#L51-L63
Jira issue: CRDB-34528
Contributor guide
Research direction
Start with pkg/server/grpc_server.go at lines 51-63 and trace the startup path and existing server modes. Define the catchup phase scope, including health checks, lease behavior, SQL connections, Raft replica work, Pebble compaction, and the bypass environment variable. Done requires an agreed design and tests for the selected startup behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100