kv/kvnemesis: TestKVNemesisMultiNode failed

kv/kvnemesis.TestKVNemesisMultiNode [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21460663?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/21460663?buildTab=artifacts#/) on master @ [ae5a328dbfc3a40bd0940b09d377a5d3ea68ea4a](https://github.com/cockroachdb/cockroach/commits/ae5a328dbfc3a40bd0940b09d377a5d3ea68ea4a):

```
=== RUN TestKVNemesisMultiNode
test_log_scope.go:172: test logs captured to: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/logTestKVNemesisMultiNode3441509696
test_log_scope.go:82: use -show-logs to present logs inline
kvnemesis_test.go:825: seed: 1373870645656810858
kvnemesis_test.go:334: n1 using clock[offset=-225.141µs,amp=3.274859ms]
kvnemesis_test.go:334: n2 using clock[offset=245.349µs,amp=3.254651ms]
kvnemesis_test.go:334: n3 using clock[offset=-64.559µs,amp=3.435441ms]
kvnemesis_test.go:334: n4 using clock[offset=335.143µs,amp=3.164857ms]
test_server_shim.go:146: using test tenant configuration from testserver factory defaults
kvnemesis_test.go:413: kvnemesis logging to /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis2618639719
kvnemesis.go:241: error applying x.AdminChangeReplicas(ctx, tk(17897218143397584135), getRangeDesc(ctx, tk(17897218143397584135), x), kvpb.ReplicationChange{ChangeType: roachpb.ADD_VOTER, Target: roachpb.ReplicationTarget{NodeID: 2, StoreID: 2}}, kvpb.ReplicationChange{ChangeType: roachpb.REMOVE_VOTER, Target: roachpb.ReplicationTarget{NodeID: 3, StoreID: 3}}) // change replicas of r87 failed: range r87:/{Table/84-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4LEARNER, next=5, gen=7] cannot make progress with proposed changes add=[(n2,s2):4LEARNER] del=[] based on live replicas []: change replicas of r87 failed: range r87:/{Table/84-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4LEARNER, next=5, gen=7] cannot make progress with proposed changes add=[(n2,s2):4LEARNER] del=[] based on live replicas []
kvnemesis.go:241: error applying x.AdminChangeReplicas(ctx, tk(552653749281840298), getRangeDesc(ctx, tk(552653749281840298), x), kvpb.ReplicationChange{ChangeType: roachpb.ADD_VOTER, Target: roachpb.ReplicationTarget{NodeID: 2, StoreID: 2}}, kvpb.ReplicationChange{ChangeType: roachpb.REMOVE_VOTER, Target: roachpb.ReplicationTarget{NodeID: 4, StoreID: 4}}) // change replicas of r87 failed: range r87:/{Table/84-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4LEARNER, next=5, gen=7] cannot make progress with proposed changes add=[(n2,s2):4LEARNER] del=[] based on live replicas []: change replicas of r87 failed: range r87:/{Table/84-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4LEARNER, next=5, gen=7] cannot make progress with proposed changes add=[(n2,s2):4LEARNER] del=[] based on live replicas []
kvnemesis.go:241: error applying x.AdminChangeReplicas(ctx, tk(7980783132572409234), getRangeDesc(ctx, tk(7980783132572409234), x), kvpb.ReplicationChange{ChangeType: roachpb.ADD_VOTER, Target: roachpb.ReplicationTarget{NodeID: 2, StoreID: 2}}) // change replicas of r87 failed: range r87:/{Table/84-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4LEARNER, next=5, gen=7] cannot make progress with proposed changes add=[(n2,s2):4LEARNER] del=[] based on live replicas []: change replicas of r87 failed: range r87:/{Table/84-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4LEARNER, next=5, gen=7] cannot make progress with proposed changes add=[(n2,s2):4LEARNER] del=[] based on live replicas []
kvnemesis.go:241: error applying x.AdminChangeReplicas(ctx, tk(10759096388226742397), getRangeDesc(ctx, tk(10759096388226742397), x), kvpb.ReplicationChange{ChangeType: roachpb.ADD_VOTER, Target: roachpb.ReplicationTarget{NodeID: 2, StoreID: 2}}) // change replicas of r87 failed: range r87:/{Table/84-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4LEARNER, next=5, gen=7] cannot make progress with proposed changes add=[(n2,s2):4LEARNER] del=[] based on live replicas []: change replicas of r87 failed: range r87:/{Table/84-Max} [(n1,s1):1, (n3,s3):2, (n4,s4):3, (n2,s2):4LEARNER, next=5, gen=7] cannot make progress with proposed changes add=[(n2,s2):4LEARNER] del=[] based on live replicas []
kvnemesis.go:282: failures(verbose): /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis2618639719/failures
repro steps: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis2618639719/repro.go
rangefeed KVs: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis2618639719/kvs-rangefeed.txt
scan KVs: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis2618639719/kvs-scan.txt
kvnemesis_test.go:1002: Metric | Node 1 | Node 2 | Node 3 | Node 4
kvnemesis_test.go:1008: ------------------------------------+--------------------------------+--------------------------------+--------------------------------+-------------------------------
kvnemesis_test.go:1015: follower_reads.success_count | 1 | 2 | 3 | 2
kvnemesis_test.go:1015: raft.commands.proposed | 3141 | 173 | 397 | 386
kvnemesis_test.go:1015: raft.commands.reproposed.new-lai | 0 | 0 | 1 | 8
kvnemesis_test.go:1015: raft.commands.reproposed.unchanged | 133 | 24 | 31 | 28
kvnemesis_test.go:1015: txn.aborts | 2687 | 1660 | 1582 | 1173
kvnemesis_test.go:1015: txn.commits | 7494 | 2465 | 2174 | 2940
kvnemesis_test.go:1015: txn.durations | μ=4ms p50=1ms p99=24ms | μ=8ms p50=2ms p99=41ms | μ=16ms p50=2ms p99=42ms | μ=19ms p50=2ms p99=88ms
kvnemesis_test.go:1015: txn.restarts.writetooold | 3 | 1 | 0 | 4
kvnemesis_test.go:1015: txn.write_buffering.disabled_after_buffering | 0 | 0 | 0 | 0
kvnemesis_test.go:1015: distsender.rpc.err.writeintenterrtype | 1 | 0 | 0 | 0
kvnemesis_test.go:1015: txn.restarts.serializable | 1 | 0 | 0 | 0
kvnemesis_test.go:1015: txn.restarts.readwithinuncertainty | 0 | 0 | 0 | 0
kvnemesis_test.go:1015: txn.restarts.commitdeadlineexceeded | 0 | 0 | 0 | 0
kvnemesis_test.go:1015: txn.server_side.1PC.success | 97 | 1 | 39 | 0
kvnemesis_test.go:1015: txnrecovery.failures | 0 | 0 | 0 | 0
kvnemesis_test.go:1015: txnrecovery.successes.aborted | 0 | 0 | 0 | 0
kvnemesis_test.go:1015: txnrecovery.successes.committed | 0 | 0 | 0 | 0
kvnemesis_test.go:1015: txnwaitqueue.deadlocks_total | 0 | 0 | 0 | 10
kvnemesis_test.go:895:
Error Trace: pkg/kv/kvnemesis/kvnemesis_test.go:895
pkg/kv/kvnemesis/kvnemesis_test.go:742
Error: Should be zero, but was 4
Test: TestKVNemesisMultiNode
Messages: kvnemesis detected failures
dump_raft_logs.go:32: dumping raft logs to /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis2618639719/raftlogs
panic.go:694: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/logTestKVNemesisMultiNode3441509696
--- FAIL: TestKVNemesisMultiNode (142.91s)
```
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #160448 kv/kvnemesis: TestKVNemesisMultiNode failed [MVCCStatsMismatch caused by PriorReadSummary on RequestLease race] [A-kv C-bug C-test-failure O-robot P-3 T-kv X-nostale branch-refs/heads/master no-test-failure-activity]

/cc @cockroachlabs/kv-triage

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66597