cockroachdb / cockroachdb/cockroach

allocator: ReplicateQueue Removing Voter Replicas Due to Disk Fullness Despite Constraint Violation

Open
#161,958 1 comment 0 reactions 0 assignees View on GitHub
A-kv A-kv-distribution A-kv-simulation branch-master C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The following asim test simulates a situation in which zone constraints are met, but disk utilization on a node exceeds 95% (`kv.allocator.max_disk_utilization_threshold`), the allocator performs load shedding despite violating constraints.

**To Reproduce**

Consider the following data driven asim test file. It creates 4 nodes. The span config mandates `s1` maintain a replica for every range. Initially there are 16 ranges of 500 MiB, store capacity is 10 GiB. Since initially `s1` contains each range, it's disk utilization is 97%.

```
gen_cluster nodes=4 region=(a,b) nodes_per_region=(1,3) store_byte_capacity_gib=10
----

gen_ranges ranges=16 repl_factor=3 placement_type=replica_placement bytes_mib=500 min_key=0 max_key=10000
{s1,s2,s3}:1 {s1,s2,s4}:1 {s1,s3,s4}:1
----

set_span_config
[0,9999999999): num_replicas=3 num_voters=3 constraints={'+region=a':1}
----

setting split_queue_enabled=false
----

eval duration=60m samples=1 seed=42 cfgs=(sma-count,mma-count) metrics=(cpu,cpu_util,leases,replicas,disk_fraction_used)
----
```

Below is a visualization from the ASIM Test Results Viewer.

Image

**Expected behavior**

No replicas should have been moved from `s1`.

**Additional Context**

It seems this issue stems from [allocator.go:1990](https://github.com/cockroachdb/cockroach/blob/1b9cc12f45dbfdbc257dd99c7ee46163acabb317/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go#L1990). The allocator correctly identifies what stores it can and cannot remove replicas from, but then it considers the set of all existing replicas, adds the potential new replica to that set and then _simulates_ "which replicas would be removed if we added a new replica for this range", disregarding the previous constraint check, and then eventually the `fullDisk` condition wins at [allocator_scorer.go:1079](https://github.com/cockroachdb/cockroach/blob/1b9cc12f45dbfdbc257dd99c7ee46163acabb317/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L1079).

Jira issue: CRDB-59195

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.