release-26.3.2-rc: roachtest: perturbation/full/partition failed

roachtest.perturbation/full/partition [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21501576?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21501576?buildTab=artifacts#/perturbation/full/partition) on release-26.3.2-rc @ [6149d98a3244ba2a3e4417d9e363da83e213e1b7](https://github.com/cockroachdb/cockroach/commits/6149d98a3244ba2a3e4417d9e363da83e213e1b7):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/perturbation/framework.go:1051
pkg/cmd/roachtest/test_runner.go:1558
src/runtime/asm_amd64.s:1771
Error: FAILURE: follower-read : throughput ratio 2.53x exceeds limit 1.2x (baseline: 10320 ops/s, measured: 4081 ops/s)
FAILURE: read : throughput ratio 2.57x exceeds limit 1.2x (baseline: 10302 ops/s, measured: 4010 ops/s)
FAILURE: write : throughput ratio 2.56x exceeds limit 1.2x (baseline: 20723 ops/s, measured: 8087 ops/s)
Test: perturbation/full/partition
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/partition/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21501576-1788584923-146-n13cpu16sm-0001 | | 10.91.100.117 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0002 | | 10.91.101.202 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0003 | | 10.91.99.15 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0004 | | 10.91.101.199 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0005 | | 10.91.99.10 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0006 | | 10.91.99.14 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0007 | | 10.91.100.115 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0008 | | 10.91.101.206 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0009 | | 10.91.101.198 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0010 | | 10.91.99.18 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0011 | | 10.91.98.114 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0012 | | 10.91.98.110 |
| teamcity-21501576-1788584923-146-n13cpu16sm-0013 | | 10.91.98.138 |

Parameters:
- acMode=defaultOption
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=16
- diskBandwidthLimit=0
- diskCount=2
- disks=2
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=expiration
- localSSD=true
- mem=standard
- numNodes=12
- numWorkloadNodes=1
- perturbationDuration=10m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=0
- splits=10000
- validationDuration=5m0s
- vcpu=16
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21501576/perturbation-full-partition/1788610998795/1788613265144)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21501576-1788584923-146-n13cpu16sm&from_ts=1788610998795&to_ts=1788613265144&live=false&storage=flex_tier)

Same failure on other branches

- #173472 release-26.3.1-rc: roachtest: perturbation/full/partition failed [A-testing C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.3.1-rc]
- #172639 release-26.3.0-rc: roachtest: perturbation/full/partition failed [C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.3.0-rc]
- #171797 release-26.3: roachtest: perturbation/full/partition failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-26.3]
- #171399 roachtest: perturbation/full/partition failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/full/partition.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67946

Start with the TeamCity logs and artifacts for perturbation/full/partition, then read pkg/cmd/roachtest/tests/perturbation/framework.go around line 1051 and pkg/cmd/roachtest/test_runner.go around line 1558. Use the roachtest README and the linked Grafana and Datadog views to investigate the throughput regression. Done means the failure cause is addressed and this test passes on the affected release branch.

Written by the indexing model from the issue text.