cockroachdb / cockroachdb/cockroach
roachtest: rebalance/by-load/replicas/mixed-version failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.
roachtest.rebalance/by-load/replicas/mixed-version [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21440170?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21440170?buildTab=artifacts#/rebalance/by-load/replicas/mixed-version) on release-25.4 @ [183b945827fc2b5034ec167f38068eff8f5772e3](https://github.com/cockroachdb/cockroach/commits/183b945827fc2b5034ec167f38068eff8f5772e3):
```
(mixedversion.go:891).Run: mixed-version test failure while running step 26 (run "rebalance load run"): CPU not evenly balanced after timeout: balanced mean=688ms/s overfull=791 (tolerance=15.0% floor=50ms/s) gap=160 (bound=238)
stores=[s1: 680ms/s (-1.1%), s2: 746ms/s (+8.4%), s3: 744ms/s (+8.2%), s4: 692ms/s (+0.6%), s5: 585ms/s (-15.0%), s6: 681ms/s (-1.1%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1
```
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=shared-process
- mvtVersions=v24.3.33 → v25.2.18 → release-25.4
- runtimeAssertionsBuild=true
- ssd=0
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21440170/rebalance-by-load-replicas-mixed-version/1785068594233/1785070511664)
/cc @cockroachlabs/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/replicas/mixed-version.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-66101
Contributor guide
Research direction
Start with the failure at mixedversion.go:891 and inspect the artifacts under /artifacts/rebalance/by-load/replicas/mixed-version/run_1, using the roachtest README for context. Compare the reported CPU-balancing timeout with the Grafana run and the mixed-version configuration. Done means identifying and fixing the cause of the failed rebalance test, then confirming the test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100