cockroachdb / cockroachdb/cockroach

roachtest: rebalance/by-load/replicas/mixed-version failed

Open
#174,847 3 comments 0 reactions 0 assignees View on GitHub
A-testing B-runtime-assertions-enabled branch-release-25.4.17-rc C-bug C-test-failure O-roachtest O-robot P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.rebalance/by-load/replicas/mixed-version [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21503676?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21503676?buildTab=artifacts#/rebalance/by-load/replicas/mixed-version) on release-25.4.17-rc @ [0447ee0a79bd81f5bcb15bcd613e6653dd0d1e0f](https://github.com/cockroachdb/cockroach/commits/0447ee0a79bd81f5bcb15bcd613e6653dd0d1e0f):

```
(mixedversion.go:891).Run: mixed-version test failure while running step 22 (run "rebalance load run"): CPU not evenly balanced after timeout: balanced (gap ok) mean=1389ms/s overfull=1597 (tolerance=15.0% floor=50ms/s) gap=334 (bound=378)
above overfull = []
below (info) = [s5: 1152ms/s (-17.0%)]
within = [s1: 1402ms/s (+1.0%), s2: 1486ms/s (+7.0%), s3: 1442ms/s (+3.9%), s4: 1456ms/s (+4.9%), s6: 1392ms/s (+0.2%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=separate-process
- mvtVersions=v25.2.23 → release-25.4.17-rc
- runtimeAssertionsBuild=true
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21503676/rebalance-by-load-replicas-mixed-version/1788865952183/1788867568315)

Same failure on other branches

- #172834 roachtest: rebalance/by-load/replicas/mixed-version failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-25.4]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/replicas/mixed-version.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Start with the failure artifacts and logs under /artifacts/rebalance/by-load/replicas/mixed-version/run_1, then inspect the mixed-version failure reported at mixedversion.go:891. Compare this run with duplicate issue #172834 and review the roachtest README and Grafana link for investigation context. Done means identifying the cause of the CPU imbalance and making the rebalance/by-load/replicas/mixed-version test pass reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.