release-26.2: roachtest: rebalance/by-load/mmc/ssds=2 failed

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.rebalance/by-load/mmc/ssds=2 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21516935?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21516935?buildTab=artifacts#/rebalance/by-load/mmc/ssds=2) on release-26.2 @ [6cbaf156f2b6f16fa3b82ee83b4939c982034d6f](https://github.com/cockroachdb/cockroach/commits/6cbaf156f2b6f16fa3b82ee83b4939c982034d6f):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/rebalance_load.go:162
pkg/cmd/roachtest/tests/rebalance_load.go:247
pkg/cmd/roachtest/test_runner.go:1601
src/runtime/asm_amd64.s:1693
Error: Received unexpected error:
CPU not evenly balanced after timeout: unbalanced mean=405ms/s overfull=466 (tolerance=15.0% floor=50ms/s) gap=306 (bound=181)
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.rebalanceByLoad.func2
| pkg/cmd/roachtest/tests/rebalance_load.go:348
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
| pkg/cmd/roachtest/roachtestutil/task/manager.go:201
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
| pkg/cmd/roachtest/roachtestutil/task/manager.go:211
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1693
Wraps: (2) CPU not evenly balanced after timeout: unbalanced mean=405ms/s overfull=466 (tolerance=15.0% floor=50ms/s) gap=306 (bound=181)
| above overfull = [s10: 516ms/s (+27.4%), s12: 469ms/s (+15.7%)]
| below (info) = [s2: 210ms/s (-48.2%), s7: 327ms/s (-19.3%)]
| within = [s1: 360ms/s (-11.0%), s3: 396ms/s (-2.1%), s4: 420ms/s (+3.7%), s5: 426ms/s (+5.1%), s6: 442ms/s (+9.1%), s8: 437ms/s (+7.9%), s9: 456ms/s (+12.7%), s11: 401ms/s (-1.0%)]
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: rebalance/by-load/mmc/ssds=2
(require.go:1401).NoError: FailNow called
test artifacts and logs in: /artifacts/rebalance/by-load/mmc/ssds=2/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21516935-1789451585-220-n7cpu4-0001 | | 10.91.97.102 |
| teamcity-21516935-1789451585-220-n7cpu4-0002 | | 10.91.97.138 |
| teamcity-21516935-1789451585-220-n7cpu4-0003 | | 10.91.97.124 |
| teamcity-21516935-1789451585-220-n7cpu4-0004 | | 10.91.97.94 |
| teamcity-21516935-1789451585-220-n7cpu4-0005 | | 10.91.96.215 |
| teamcity-21516935-1789451585-220-n7cpu4-0006 | | 10.91.97.100 |
| teamcity-21516935-1789451585-220-n7cpu4-0007 | | 10.91.97.89 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=2
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21516935/rebalance-by-load-mmc-ssds-2/1789479744218/1789480893871)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*rebalance/by-load/mmc/ssds=2.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68425

Start with pkg/cmd/roachtest/tests/rebalance_load.go, especially the failure paths at lines 162, 247, and 348, then inspect the linked TeamCity logs, artifacts, and Grafana run for rebalance/by-load/mmc/ssds=2. Reproduce the roachtest if possible and determine why CPU remains unbalanced; done means the failure is understood and the test no longer reports this unexpected imbalance.

Written by the indexing model from the issue text.