cockroachdb / cockroachdb/cockroach

storage: disk stall during decommission or addNode test with large values

Open
#133,606 3 comments 0 reactions 1 assignee Claimed by @angeladietz View on GitHub
A-admission-control branch-master C-bug O-perturbation T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The test `perturbation/metamorphic/decommission` will fail with large block sizes. This manifests as a disk stall on one of the non-decommissioning nodes.

**To Reproduce**

Run the test `perturbation/metamorphic/decommission` with a patch to increase the maxBlockBytes, with a small node count.
```
diff --git a/pkg/cmd/roachtest/tests/admission_control_latency.go b/pkg/cmd/roachtest/tests/admission_control_latency.go
index b9d96fb3f1e..785e8c9ecca 100644
--- a/pkg/cmd/roachtest/tests/admission_control_latency.go
+++ b/pkg/cmd/roachtest/tests/admission_control_latency.go
@@ -619,7 +619,10 @@ var _ perturbation = decommission{}

func (d decommission) setupMetamorphic(rng *rand.Rand) variations {
d.drain = rng.Intn(2) == 0
- return newMetamorphic(d, rng)
+ v := newMetamorphic(d, rng)
+ v.numNodes = 5
+ v.maxBlockBytes = 4096
+ v.cpu = 16
+ v.disks = 1
+ return v
}
```

Notice that the test will fail due to the high latency impact of the decommission.

**Expected behavior**
A decommission should not impact latency of normal operations.

**Additional data / screenshots**
See https://github.com/cockroachdb/cockroach/issues/133001#issuecomment-2441467411 for a reproduction.

The test will pass with either more nodes or a smaller block size. For now this configuration has been disabled in the test.

Jira issue: CRDB-43705

Epic CRDB-63409

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.