release-26.3.2-rc: roachtest: perturbation/full/splits/asleep=true failed

roachtest.perturbation/full/splits/asleep=true [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21520640?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21520640?buildTab=artifacts#/perturbation/full/splits/asleep=true) on release-26.3.2-rc @ [8db696c79524f16c5fd046165e52a2e8c7eff326](https://github.com/cockroachdb/cockroach/commits/8db696c79524f16c5fd046165e52a2e8c7eff326):

Failed with:

```
(soon.go:64).SucceedsWithin: condition failed to evaluate within 1h0m0s: from validation_check.go:421: tokens not returned for kvflowcontrol.tokens.send.elastic: deducted 130489919498 returned 130489890826
goroutine dump: /tmp/goroutine_dump_452025008.txt
test artifacts and logs in: /artifacts/perturbation/full/splits/asleep=true/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21520640-1789621727-171-n13cpu16sm-0001 | | 10.91.97.101 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0002 | | 10.91.98.140 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0003 | | 10.91.97.92 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0004 | | 10.91.98.61 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0005 | | 10.91.97.113 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0006 | | 10.91.98.29 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0007 | | 10.91.98.129 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0008 | | 10.91.98.36 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0009 | | 10.91.97.129 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0010 | | 10.91.97.34 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0011 | | 10.91.97.15 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0012 | | 10.91.97.27 |
| teamcity-21520640-1789621727-171-n13cpu16sm-0013 | | 10.91.97.28 |

Parameters:
- acMode=defaultOption
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=16
- diskBandwidthLimit=0
- diskCount=2
- disks=2
- encrypted=false
- fillDuration=10m0s
- leaseType=leader
- mem=standard
- numNodes=12
- numWorkloadNodes=1
- perturbationDuration=10m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=0
- splits=500
- validationDuration=5m0s
- vcpu=16
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21520640/perturbation-full-splits-asleep-true/1789647686276/1789660787547)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21520640-1789621727-171-n13cpu16sm&from_ts=1789647686276&to_ts=1789660787547&live=false&storage=flex_tier)

Same failure on other branches

- #174942 basalt-26.3: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-basalt-26.3]
- #173742 release-26.3.1-rc: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-26.3.1-rc]
- #171839 release-26.3: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-26.3]
- #171278 roachtest: perturbation/full/splits/asleep=true failed [C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-master]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/full/splits/asleep=true.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68477

Start with the roachtest README and the failure references in soon.go:64 and validation_check.go:421. Inspect the linked TeamCity artifacts, logs, Grafana, and Datadog data for perturbation/full/splits/asleep=true, then compare the repeated failures in issues #174942, #173742, #171839, and #171278. Done means identifying the cause of the missing kvflowcontrol tokens and preventing or reliably reproducing the failure.

Written by the indexing model from the issue text.