cockroachdb / cockroachdb/cockroach
storage: disk stall during decommission or addNode test with large values
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The test `perturbation/metamorphic/decommission` will fail with large block sizes. This manifests as a disk stall on one of the non-decommissioning nodes.
**To Reproduce**
Run the test `perturbation/metamorphic/decommission` with a patch to increase the maxBlockBytes, with a small node count.
```
diff --git a/pkg/cmd/roachtest/tests/admission_control_latency.go b/pkg/cmd/roachtest/tests/admission_control_latency.go
index b9d96fb3f1e..785e8c9ecca 100644
--- a/pkg/cmd/roachtest/tests/admission_control_latency.go
+++ b/pkg/cmd/roachtest/tests/admission_control_latency.go
@@ -619,7 +619,10 @@ var _ perturbation = decommission{}
func (d decommission) setupMetamorphic(rng *rand.Rand) variations {
d.drain = rng.Intn(2) == 0
- return newMetamorphic(d, rng)
+ v := newMetamorphic(d, rng)
+ v.numNodes = 5
+ v.maxBlockBytes = 4096
+ v.cpu = 16
+ v.disks = 1
+ return v
}
```
Notice that the test will fail due to the high latency impact of the decommission.
**Expected behavior**
A decommission should not impact latency of normal operations.
**Additional data / screenshots**
See https://github.com/cockroachdb/cockroach/issues/133001#issuecomment-2441467411 for a reproduction.
The test will pass with either more nodes or a smaller block size. For now this configuration has been disabled in the test.
Jira issue: CRDB-43705
Epic CRDB-63409
Contributor guide
Assessment
This issue has not been assessed yet.