cockroachdb / cockroachdb/cockroach
roachtest: rebalance/by-load/mode=mmc failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.rebalance/by-load/mode=mmc [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyIbmBazel/21431550?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyIbmBazel/21431550?buildTab=artifacts#/rebalance/by-load/mode=mmc) on release-26.1 @ [bd5f20c106a38196fa56e2e7febcc12bcfb4addc](https://github.com/cockroachdb/cockroach/commits/bd5f20c106a38196fa56e2e7febcc12bcfb4addc):
Failed with:
```
(assertions.go:369).Fail:
Error Trace: pkg/cmd/roachtest/tests/rebalance_load.go:162
pkg/cmd/roachtest/tests/rebalance_load.go:247
pkg/cmd/roachtest/test_runner.go:1578
src/runtime/asm_amd64.s:1693
Error: Received unexpected error:
CPU not evenly balanced after timeout: unbalanced mean=388ms/s overfull=446 (tolerance=15.0% floor=50ms/s) gap=236 (bound=178)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.rebalanceByLoad.func2
| pkg/cmd/roachtest/tests/rebalance_load.go:348
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
| pkg/cmd/roachtest/roachtestutil/task/manager.go:201
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
| pkg/cmd/roachtest/roachtestutil/task/manager.go:211
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (2) CPU not evenly balanced after timeout: unbalanced mean=388ms/s overfull=446 (tolerance=15.0% floor=50ms/s) gap=236 (bound=178)
| above overfull = [s3: 491ms/s (+26.6%), s6: 488ms/s (+26.1%)]
| below (info) = [s1: 255ms/s (-34.2%), s2: 326ms/s (-15.8%)]
| within = [s4: 376ms/s (-2.9%), s5: 388ms/s (+0.2%)]
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: rebalance/by-load/mode=mmc
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/rebalance/by-load/mode=mmc/cpu_arch=s390x/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21431550-1784609368-101-n7cpu4-0001 | 158.176.170.101 | 10.242.128.190 |
| teamcity-21431550-1784609368-101-n7cpu4-0002 | 158.176.178.81 | 10.242.128.186 |
| teamcity-21431550-1784609368-101-n7cpu4-0003 | 158.176.193.51 | 10.242.128.184 |
| teamcity-21431550-1784609368-101-n7cpu4-0004 | 158.176.190.200 | 10.242.128.187 |
| teamcity-21431550-1784609368-101-n7cpu4-0005 | 158.176.178.51 | 10.242.128.189 |
| teamcity-21431550-1784609368-101-n7cpu4-0006 | 158.176.194.182 | 10.242.128.185 |
| teamcity-21431550-1784609368-101-n7cpu4-0007 | 158.176.175.116 | 10.242.128.188 |
Parameters:
- arch=s390x
- cloud=ibm
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- metamorphicLeases=default
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
_Grafana is not yet available for ibm clusters_
/cc @cockroachlabs/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/mode=mmc.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-65948
Contributor guide
Assessment
This issue has not been assessed yet.