cockroachdb / cockroachdb/cockroach
release-26.2: roachtest: rebalance/by-load/mmc failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.rebalance/by-load/mmc [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21433345?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21433345?buildTab=artifacts#/rebalance/by-load/mmc) on release-26.2 @ [2cf3b002ec79a6057831532053e886741ea507ed](https://github.com/cockroachdb/cockroach/commits/2cf3b002ec79a6057831532053e886741ea507ed):
Failed with:
```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/rebalance_load.go:162
pkg/cmd/roachtest/tests/rebalance_load.go:247
pkg/cmd/roachtest/test_runner.go:1591
src/runtime/asm_amd64.s:1693
Error: Received unexpected error:
CPU not evenly balanced after timeout: unbalanced mean=600ms/s overfull=690 (tolerance=15.0% floor=50ms/s) gap=463 (bound=220)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.rebalanceByLoad.func2
| pkg/cmd/roachtest/tests/rebalance_load.go:348
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
| pkg/cmd/roachtest/roachtestutil/task/manager.go:201
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
| pkg/cmd/roachtest/roachtestutil/task/manager.go:211
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (2) CPU not evenly balanced after timeout: unbalanced mean=600ms/s overfull=690 (tolerance=15.0% floor=50ms/s) gap=463 (bound=220)
| above overfull = [s2: 691ms/s (+15.2%)]
| below (info) = [s1: 228ms/s (-61.9%)]
| within = [s3: 680ms/s (+13.3%), s4: 655ms/s (+9.3%), s5: 675ms/s (+12.5%), s6: 669ms/s (+11.5%)]
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: rebalance/by-load/mmc
(require.go:1401).NoError: FailNow called
test artifacts and logs in: /artifacts/rebalance/by-load/mmc/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21433345-1784725796-196-n7cpu4-0001 | 34.75.12.100 | 10.142.2.80 |
| teamcity-21433345-1784725796-196-n7cpu4-0002 | 35.243.243.150 | 10.142.2.62 |
| teamcity-21433345-1784725796-196-n7cpu4-0003 | 35.196.183.159 | 10.142.2.21 |
| teamcity-21433345-1784725796-196-n7cpu4-0004 | 35.229.71.64 | 10.142.2.75 |
| teamcity-21433345-1784725796-196-n7cpu4-0005 | 34.73.173.21 | 10.142.2.65 |
| teamcity-21433345-1784725796-196-n7cpu4-0006 | 34.73.230.60 | 10.142.1.246 |
| teamcity-21433345-1784725796-196-n7cpu4-0007 | 34.24.97.65 | 10.142.2.22 |
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21433345/rebalance-by-load-mmc/1784753959918/1784755116027)
/cc @cockroachlabs/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/mmc.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65996
Contributor guide
Assessment
This issue has not been assessed yet.