cockroachdb / cockroachdb/cockroach

allocator: prevent multiple replicas from sharing a host

Open
#153,863 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

On high-CPU machines, to avoid running CRDB processes across NUMA sockets, customers want to run multiple incarnations of CRDB on a single machine. This raises the question of fault tolerance: since a machine is a failure domain, in general we never want two replicas to reside on two nodes that share an underlying machine.

By using locality tags (for example `region=x,...,host=X`) the allocator "usually" will avoid placing two replicas on nodes with matching `host=X` tag, but it's still possible that it sometimes may end up doing that, as locality diversity is strictly speaking "best effort".

We have an existing check that can prevent a new replica to be allocated on a node that has an existing replica,

https://github.com/cockroachdb/cockroach/blob/a1ef101368f567190c4b49561330ea6b4a0f5782/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L1207-L1211

but this does not apply to the case in which multiple nodes run on a single machine (as each would have their own NodeID, and the allocator wouldn't realize they're on the same node).

[Slack](https://cockroachlabs.slack.com/archives/C0KB9Q03D/p1752788949654439)

Epic CRDB-56265
Jira issue: CRDB-54642

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.