cockroachdb / cockroachdb/cockroach

roachtest: rebalance/by-load/replicas/mixed-version failed

Open
#172,834 4 comments 0 reactions 0 assignees View on GitHub
A-testing B-runtime-assertions-enabled branch-release-25.4 C-bug C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.rebalance/by-load/replicas/mixed-version [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21440170?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21440170?buildTab=artifacts#/rebalance/by-load/replicas/mixed-version) on release-25.4 @ [183b945827fc2b5034ec167f38068eff8f5772e3](https://github.com/cockroachdb/cockroach/commits/183b945827fc2b5034ec167f38068eff8f5772e3):

```
(mixedversion.go:891).Run: mixed-version test failure while running step 26 (run "rebalance load run"): CPU not evenly balanced after timeout: balanced mean=688ms/s overfull=791 (tolerance=15.0% floor=50ms/s) gap=160 (bound=238)
stores=[s1: 680ms/s (-1.1%), s2: 746ms/s (+8.4%), s3: 744ms/s (+8.2%), s4: 692ms/s (+0.6%), s5: 585ms/s (-15.0%), s6: 681ms/s (-1.1%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1
```

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=shared-process
- mvtVersions=v24.3.33 → v25.2.18 → release-25.4
- runtimeAssertionsBuild=true
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21440170/rebalance-by-load-replicas-mixed-version/1785068594233/1785070511664)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/replicas/mixed-version.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66101

Contributor guide

Open the contributing guide

Research direction

Start with the failure at mixedversion.go:891 and inspect the artifacts under /artifacts/rebalance/by-load/replicas/mixed-version/run_1, using the roachtest README for context. Compare the reported CPU-balancing timeout with the Grafana run and the mixed-version configuration. Done means identifying and fixing the cause of the failed rebalance test, then confirming the test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.