cockroachdb / cockroachdb/cockroach
release-26.2: roachtest: rebalance/by-load/mmc/ssds=2 failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.
roachtest.rebalance/by-load/mmc/ssds=2 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21467550?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21467550?buildTab=artifacts#/rebalance/by-load/mmc/ssds=2) on release-26.2 @ [4d113865cfbe2c260ff5bf56af1f781f2f2160dd](https://github.com/cockroachdb/cockroach/commits/4d113865cfbe2c260ff5bf56af1f781f2f2160dd):
Failed with:
```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/rebalance_load.go:162
pkg/cmd/roachtest/tests/rebalance_load.go:247
pkg/cmd/roachtest/test_runner.go:1601
src/runtime/asm_amd64.s:1693
Error: Received unexpected error:
CPU not evenly balanced after timeout: unbalanced mean=399ms/s overfull=458 (tolerance=15.0% floor=50ms/s) gap=258 (bound=180)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.rebalanceByLoad.func2
| pkg/cmd/roachtest/tests/rebalance_load.go:348
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
| pkg/cmd/roachtest/roachtestutil/task/manager.go:201
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
| pkg/cmd/roachtest/roachtestutil/task/manager.go:211
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (2) CPU not evenly balanced after timeout: unbalanced mean=399ms/s overfull=458 (tolerance=15.0% floor=50ms/s) gap=258 (bound=180)
| above overfull = [s6: 517ms/s (+29.7%)]
| below (info) = [s7: 331ms/s (-17.1%), s8: 259ms/s (-35.0%)]
| within = [s1: 405ms/s (+1.6%), s2: 412ms/s (+3.3%), s3: 452ms/s (+13.5%), s4: 345ms/s (-13.5%), s5: 401ms/s (+0.5%), s9: 432ms/s (+8.4%), s10: 434ms/s (+9.0%), s11: 419ms/s (+5.2%), s12: 376ms/s (-5.7%)]
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: rebalance/by-load/mmc/ssds=2
(require.go:1401).NoError: FailNow called
test artifacts and logs in: /artifacts/rebalance/by-load/mmc/ssds=2/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21467550-1786858750-373-n7cpu4-0001 | | 10.91.96.19 |
| teamcity-21467550-1786858750-373-n7cpu4-0002 | | 10.91.96.85 |
| teamcity-21467550-1786858750-373-n7cpu4-0003 | | 10.91.96.18 |
| teamcity-21467550-1786858750-373-n7cpu4-0004 | | 10.91.96.54 |
| teamcity-21467550-1786858750-373-n7cpu4-0005 | | 10.91.96.6 |
| teamcity-21467550-1786858750-373-n7cpu4-0006 | | 10.91.96.14 |
| teamcity-21467550-1786858750-373-n7cpu4-0007 | | 10.91.96.47 |
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=2
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=true
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21467550/rebalance-by-load-mmc-ssds-2/1786912243577/1786913395598)
/cc @cockroachlabs/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/mmc/ssds=2.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-66740
Contributor guide
Assessment
This issue has not been assessed yet.