roachtest: rebalance/by-load/mode=mmc/ssds=2 failed

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.rebalance/by-load/mode=mmc/ssds=2 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21470037?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21470037?buildTab=artifacts#/rebalance/by-load/mode=mmc/ssds=2) on release-26.1 @ [3b741e466ca90d6c5b293232ad72a000346058d3](https://github.com/cockroachdb/cockroach/commits/3b741e466ca90d6c5b293232ad72a000346058d3):

Failed with:

```
(assertions.go:369).Fail:
Error Trace: pkg/cmd/roachtest/tests/rebalance_load.go:162
pkg/cmd/roachtest/tests/rebalance_load.go:247
pkg/cmd/roachtest/test_runner.go:1588
src/runtime/asm_amd64.s:1693
Error: Received unexpected error:
CPU not evenly balanced after timeout: unbalanced mean=510ms/s overfull=586 (tolerance=15.0% floor=50ms/s) gap=297 (bound=202)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.rebalanceByLoad.func2
| pkg/cmd/roachtest/tests/rebalance_load.go:348
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
| pkg/cmd/roachtest/roachtestutil/task/manager.go:201
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
| pkg/cmd/roachtest/roachtestutil/task/manager.go:211
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (2) CPU not evenly balanced after timeout: unbalanced mean=510ms/s overfull=586 (tolerance=15.0% floor=50ms/s) gap=297 (bound=202)
| above overfull = [s9: 606ms/s (+18.9%), s10: 647ms/s (+26.9%)]
| below (info) = [s1: 405ms/s (-20.6%), s2: 350ms/s (-31.3%)]
| within = [s3: 585ms/s (+14.8%), s4: 530ms/s (+4.0%), s5: 455ms/s (-10.7%), s6: 521ms/s (+2.2%), s7: 455ms/s (-10.8%), s8: 442ms/s (-13.2%), s11: 547ms/s (+7.3%), s12: 574ms/s (+12.6%)]
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: rebalance/by-load/mode=mmc/ssds=2
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/rebalance/by-load/mode=mmc/ssds=2/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21470037-1787035394-362-n7cpu4-0001 | | 10.91.96.69 |
| teamcity-21470037-1787035394-362-n7cpu4-0002 | | 10.91.96.204 |
| teamcity-21470037-1787035394-362-n7cpu4-0003 | | 10.91.96.147 |
| teamcity-21470037-1787035394-362-n7cpu4-0004 | | 10.91.96.98 |
| teamcity-21470037-1787035394-362-n7cpu4-0005 | | 10.91.96.139 |
| teamcity-21470037-1787035394-362-n7cpu4-0006 | | 10.91.96.83 |
| teamcity-21470037-1787035394-362-n7cpu4-0007 | | 10.91.96.116 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=2
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=leader
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21470037/rebalance-by-load-mode-mmc-ssds-2/1787082685814/1787083819532)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/mode=mmc/ssds=2.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66796

Start with pkg/cmd/roachtest/tests/rebalance_load.go at lines 162, 247, and 348, then inspect the linked TeamCity artifacts and logs for rebalance/by-load/mode=mmc/ssds=2. Re-run that named roachtest and determine why CPU balancing times out; done means the test reliably reaches an even balance without the reported failure.

Written by the indexing model from the issue text.