cockroachdb / cockroachdb/cockroach
roachtest: rebalance/by-load/replicas/mixed-version failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.
roachtest.rebalance/by-load/replicas/mixed-version [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21503676?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21503676?buildTab=artifacts#/rebalance/by-load/replicas/mixed-version) on release-25.4.17-rc @ [0447ee0a79bd81f5bcb15bcd613e6653dd0d1e0f](https://github.com/cockroachdb/cockroach/commits/0447ee0a79bd81f5bcb15bcd613e6653dd0d1e0f):
```
(mixedversion.go:891).Run: mixed-version test failure while running step 22 (run "rebalance load run"): CPU not evenly balanced after timeout: balanced (gap ok) mean=1389ms/s overfull=1597 (tolerance=15.0% floor=50ms/s) gap=334 (bound=378)
above overfull = []
below (info) = [s5: 1152ms/s (-17.0%)]
within = [s1: 1402ms/s (+1.0%), s2: 1486ms/s (+7.0%), s3: 1442ms/s (+3.9%), s4: 1456ms/s (+4.9%), s6: 1392ms/s (+0.2%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1
```
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=separate-process
- mvtVersions=v25.2.23 → release-25.4.17-rc
- runtimeAssertionsBuild=true
- ssd=0
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21503676/rebalance-by-load-replicas-mixed-version/1788865952183/1788867568315)
Same failure on other branches
- #172834 roachtest: rebalance/by-load/replicas/mixed-version failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-25.4]
/cc @cockroachlabs/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/replicas/mixed-version.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Contributor guide
Research direction
Start with the failure artifacts and logs under /artifacts/rebalance/by-load/replicas/mixed-version/run_1, then inspect the mixed-version failure reported at mixedversion.go:891. Compare this run with duplicate issue #172834 and review the roachtest README and Grafana link for investigation context. Done means identifying the cause of the CPU imbalance and making the rebalance/by-load/replicas/mixed-version test pass reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100