cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: perturbation/full/splits/asleep=true failed

Open
#171,839 27 comments 0 reactions 1 assignee Claimed by @miraradeva View on GitHub
branch-release-26.3 C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.perturbation/full/splits/asleep=true [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21394403?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21394403?buildTab=artifacts#/perturbation/full/splits/asleep=true) on release-26.3 @ [9caafebcafcd887c18107a3a718e5a07f38938d0](https://github.com/cockroachdb/cockroach/commits/9caafebcafcd887c18107a3a718e5a07f38938d0):

Failed with:

```
(splits.go:345).startPerturbation: splits[splitperf_gn7s1_n8s1_n9s1.t]: lease rebalance failure rate 503/682 = 74% exceeds 10%; first error: dial tcp 35.229.90.155:26257: connect: connection refused
test artifacts and logs in: /artifacts/perturbation/full/splits/asleep=true/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21394403-1781935825-331-n13cpu16sm-0001 | 35.229.90.155 | 10.142.0.124 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0002 | 34.139.120.251 | 10.142.0.251 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0003 | 34.74.132.146 | 10.142.0.254 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0004 | 34.73.178.92 | 10.142.0.253 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0005 | 34.73.134.210 | 10.142.0.247 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0006 | 34.138.187.209 | 10.142.0.255 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0007 | 34.24.76.217 | 10.142.0.252 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0008 | 34.26.160.53 | 10.142.0.132 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0009 | 34.138.42.81 | 10.142.0.246 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0010 | 35.196.224.193 | 10.142.0.193 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0011 | 34.24.241.155 | 10.142.1.0 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0012 | 34.75.147.249 | 10.142.0.184 |
| teamcity-21394403-1781935825-331-n13cpu16sm-0013 | 34.26.192.13 | 10.142.0.250 |

Parameters:
- acMode=defaultOption
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=16
- diskBandwidthLimit=0
- diskCount=2
- disks=2
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=leader
- localSSD=true
- mem=standard
- numNodes=12
- numWorkloadNodes=1
- perturbationDuration=10m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=0
- splits=500
- validationDuration=5m0s
- vcpu=16
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21394403/perturbation-full-splits-asleep-true/1782010793069/1782015014920)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21394403-1781935825-331-n13cpu16sm&from_ts=1782010793069&to_ts=1782015014920&live=false&storage=flex_tier)

/cc @cockroachdb/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/full/splits/asleep=true.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-64975

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.