roachtest: perturbation/full/splits/asleep=false failed

Nobody has claimed this yet.

roachtest.perturbation/full/splits/asleep=false failed with artifacts on master @ 6ca2bac2989e7c037f7853a7a69333097c769372:

Failed with:

(splits.go:243).startPerturbation: query Cockroach node ID: dial tcp 10.91.96.125:26257: connect: connection refused
test artifacts and logs in: /artifacts/perturbation/full/splits/asleep=false/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21523590-1789797357-179-n13cpu16sm-0001		10.91.96.201
teamcity-21523590-1789797357-179-n13cpu16sm-0002		10.91.96.197
teamcity-21523590-1789797357-179-n13cpu16sm-0003		10.91.96.240
teamcity-21523590-1789797357-179-n13cpu16sm-0004		10.91.96.204
teamcity-21523590-1789797357-179-n13cpu16sm-0005		10.91.97.15
teamcity-21523590-1789797357-179-n13cpu16sm-0006		10.91.96.153
teamcity-21523590-1789797357-179-n13cpu16sm-0007		10.91.96.220
teamcity-21523590-1789797357-179-n13cpu16sm-0008		10.91.96.178
teamcity-21523590-1789797357-179-n13cpu16sm-0009		10.91.96.186
teamcity-21523590-1789797357-179-n13cpu16sm-0010		10.91.97.21
teamcity-21523590-1789797357-179-n13cpu16sm-0011		10.91.96.143
teamcity-21523590-1789797357-179-n13cpu16sm-0012		10.91.96.26
teamcity-21523590-1789797357-179-n13cpu16sm-0013		10.91.96.125

Parameters:

acMode=defaultOption
arch=amd64
blockSize=4096
cloud=gce
concurrency=500
coverageBuild=false
cpu=16
diskBandwidthLimit=0
diskCount=2
disks=2
encrypted=false
fillDuration=10m0s
followerReadPercent=50
fs=ext4
leaseType=leader
localSSD=true
mem=standard
numNodes=12
numWorkloadNodes=1
perturbationDuration=10m0s
ratioOfMax=0.5
readPercent=50
runtimeAssertionsBuild=false
seed=0
splits=500
validationDuration=5m0s
vcpu=16
workloadConnInit=

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

Same failure on other branches

#175662 release-26.4: roachtest: perturbation/full/splits/asleep=false failed [C-test-failure O-roachtest O-robot T-kv branch-release-26.4 release-blocker]

/cc @cockroachlabs/kv-triage _{Improve this report!}

Jira issue: CRDB-68563

Start with the failure at splits.go:243 and inspect the artifacts and logs under /artifacts/perturbation/full/splits/asleep=false/run_1. Read the roachtest README, then compare the Grafana and Datadog data for node 10.91.96.125 and the reported connection refusal. Done means identifying the cause and making this perturbation test complete successfully.

Written by the indexing model from the issue text.

Description

Contributor guide

First steps

Research direction

Assessment