cockroachdb / cockroachdb/cockroach
roachtest: perturbation/full/splits/asleep=true failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.perturbation/full/splits/asleep=true [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21366438?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21366438?buildTab=artifacts#/perturbation/full/splits/asleep=true) on master @ [2fffba777c3a77451a6480b09316d0ae985f0b48](https://github.com/cockroachdb/cockroach/commits/2fffba777c3a77451a6480b09316d0ae985f0b48):
Failed with:
```
(splits.go:317).startPerturbation: split splitperf_gn1s2_n2s2_n3s2.t batch 55: pq: replica unavailable: (n3,s6):1 unable to serve request to r540369:/Table/12{7/1/9221896317005139107-8} [(n3,s6):1, (n2,s4):2, (n1,s2):3, next=4, gen=179, sticky=9223372036.854775807,2147483647]: closed timestamp: 1780247439.453188868,0 (2026-05-31 17:10:39); raft status: {"id":"1","term":15,"vote":"3","commit":14,"lead":"0","leadEpoch":"0","raftState":"StatePreCandidate","applied":14,"progress":{},"leadtransferee":"0"}: replica has been leaderless for 1m0s
test artifacts and logs in: /artifacts/perturbation/full/splits/asleep=true/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21366438-1780208222-176-n13cpu16sm-0001 | 34.26.121.159 | 10.142.1.39 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0002 | 34.74.244.171 | 10.142.0.241 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0003 | 35.185.8.165 | 10.142.1.44 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0004 | 34.74.121.97 | 10.142.0.156 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0005 | 35.190.168.71 | 10.142.0.214 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0006 | 34.24.202.240 | 10.142.1.17 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0007 | 35.237.236.250 | 10.142.0.218 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0008 | 34.24.124.183 | 10.142.0.178 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0009 | 35.196.152.39 | 10.142.1.22 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0010 | 34.75.135.13 | 10.142.0.239 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0011 | 104.196.183.24 | 10.142.1.38 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0012 | 34.74.143.164 | 10.142.1.144 |
| teamcity-21366438-1780208222-176-n13cpu16sm-0013 | 104.196.162.238 | 10.142.0.237 |
Parameters:
- acMode=defaultOption
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=16
- diskBandwidthLimit=0
- diskCount=2
- disks=2
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=leader
- localSSD=true
- mem=standard
- numNodes=12
- numWorkloadNodes=1
- perturbationDuration=10m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=0
- splits=500
- validationDuration=5m0s
- vcpu=16
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21366438/perturbation-full-splits-asleep-true/1780245269118/1780248164983)
See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21366438-1780208222-176-n13cpu16sm&from_ts=1780245269118&to_ts=1780248164983&live=false&storage=flex_tier)
/cc @cockroachdb/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/full/splits/asleep=true.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-64417
Contributor guide
Assessment
This issue has not been assessed yet.