roachtest: perturbation/full/backup failed

roachtest.perturbation/full/backup [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21419925?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21419925?buildTab=artifacts#/perturbation/full/backup) on release-26.1 @ [61439c4e5863f27019998a4a8aba2ce642492f34](https://github.com/cockroachdb/cockroach/commits/61439c4e5863f27019998a4a8aba2ce642492f34):

Failed with:

```
(assertions.go:369).Fail:
Error Trace: pkg/cmd/roachtest/tests/perturbation/framework.go:717
pkg/cmd/roachtest/test_runner.go:1578
src/runtime/asm_amd64.s:1693
Error: Should be true
Test: perturbation/full/backup
Messages: FAILURE: follower-read : Increase 537774.9561 > 5.0000 BASE: 6.69425ms SCORE: 1h0m0s

FAILURE: read : Increase 541771.7290 > 5.0000 BASE: 6.644865ms SCORE: 1h0m0s

FAILURE: write : Increase 483733.2598 > 5.0000 BASE: 7.442118ms SCORE: 1h0m0s
(require.go:2014).True: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/backup/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21419925-1783919346-181-n13cpu16sm-0001 | 34.23.196.146 | 10.142.1.7 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0002 | 35.231.188.198 | 10.142.1.215 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0003 | 35.243.150.146 | 10.142.0.203 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0004 | 34.148.35.17 | 10.142.0.112 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0005 | 35.196.193.203 | 10.142.0.228 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0006 | 34.75.245.146 | 10.142.1.239 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0007 | 34.148.20.0 | 10.142.1.224 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0008 | 35.229.43.35 | 10.142.1.212 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0009 | 104.196.219.0 | 10.142.0.177 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0010 | 34.23.92.102 | 10.142.1.217 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0011 | 35.237.18.114 | 10.142.1.111 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0012 | 35.243.161.210 | 10.142.0.180 |
| teamcity-21419925-1783919346-181-n13cpu16sm-0013 | 35.185.109.240 | 10.142.1.149 |

Parameters:
- acMode=defaultOption
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=16
- diskBandwidthLimit=0
- diskCount=2
- disks=2
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=leader
- localSSD=true
- mem=standard
- numNodes=12
- numWorkloadNodes=1
- perturbationDuration=10m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=0
- splits=10000
- validationDuration=5m0s
- vcpu=16
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21419925/perturbation-full-backup/1783947237559/1783948957064)

Same failure on other branches

- #171256 roachtest: perturbation/full/backup failed [A-testing C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.1.5-rc p3]
- #171157 roachtest: perturbation/full/backup failed [C-test-failure O-perturbation O-roachtest O-robot P-3 T-kv X-duplicate branch-release-25.4.11-rc]
- #170480 roachtest: perturbation/full/backup failed [A-testing C-bug C-test-failure O-perturbation O-roachtest O-robot P-3 T-kv X-duplicate branch-release-25.4]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/full/backup.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65727

Start with the failure artifacts at /artifacts/perturbation/full/backup/run_1 and the Grafana link, then read pkg/cmd/roachtest/tests/perturbation/framework.go around line 717 and the roachtest README. Compare the repeated failures linked in the issue and determine the cause of the latency increase; done means perturbation/full/backup passes on the affected branch without masking the reported validation failures.

Written by the indexing model from the issue text.