release-26.4: roachtest: perturbation/full/splits/as... | cockroachdb/cockroach

Nobody has claimed this yet.

roachtest.perturbation/full/splits/asleep=true failed with artifacts on release-26.4 @ b4884d371d74d172ab184bc4ef0924ccb758f3c2:

Failed with:

(splits.go:243).startPerturbation: query Cockroach node ID: dial tcp 10.91.101.80:26257: connect: connection refused
test artifacts and logs in: /artifacts/perturbation/full/splits/asleep=true/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21523842-1789803587-02-n13cpu16sm-0001		10.91.101.150
teamcity-21523842-1789803587-02-n13cpu16sm-0002		10.91.101.161
teamcity-21523842-1789803587-02-n13cpu16sm-0003		10.91.101.149
teamcity-21523842-1789803587-02-n13cpu16sm-0004		10.91.101.138
teamcity-21523842-1789803587-02-n13cpu16sm-0005		10.91.101.151
teamcity-21523842-1789803587-02-n13cpu16sm-0006		10.91.101.152
teamcity-21523842-1789803587-02-n13cpu16sm-0007		10.91.101.148
teamcity-21523842-1789803587-02-n13cpu16sm-0008		10.91.101.168
teamcity-21523842-1789803587-02-n13cpu16sm-0009		10.91.101.169
teamcity-21523842-1789803587-02-n13cpu16sm-0010		10.91.101.139
teamcity-21523842-1789803587-02-n13cpu16sm-0011		10.91.101.74
teamcity-21523842-1789803587-02-n13cpu16sm-0012		10.91.101.79
teamcity-21523842-1789803587-02-n13cpu16sm-0013		10.91.101.80

Parameters:

acMode=defaultOption
arch=amd64
blockSize=4096
cloud=gce
concurrency=500
coverageBuild=false
cpu=16
diskBandwidthLimit=0
diskCount=2
disks=2
encrypted=false
fillDuration=10m0s
followerReadPercent=50
leaseType=leader
mem=standard
numNodes=12
numWorkloadNodes=1
perturbationDuration=10m0s
ratioOfMax=0.5
readPercent=50
runtimeAssertionsBuild=false
seed=0
splits=500
validationDuration=5m0s
vcpu=16
workloadConnInit=

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

See: Datadog Logs

Same failure on other branches

#175589 release-26.3.2-rc: roachtest: perturbation/full/splits/asleep=true failed [A-kv C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-26.3.2-rc]
#174942 basalt-26.3: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-basalt-26.3]
#173742 release-26.3.1-rc: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-26.3.1-rc]
#171839 release-26.3: roachtest: perturbation/full/splits/asleep=true failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-26.3]
#171278 roachtest: perturbation/full/splits/asleep=true failed [C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-master]

/cc @cockroachlabs/kv-triage _{Improve this report!}

Jira issue: CRDB-68549

Start at splits.go:243, where startPerturbation reports the failed node-ID query, then inspect the linked TeamCity logs and artifacts under /artifacts/perturbation/full/splits/asleep=true/run_1. Compare the repeated failures listed in this issue and use the roachtest README for test context. Done means identifying and correcting the cause of the connection refusal, with the affected roachtest passing.

Written by the indexing model from the issue text.

release-26.4: roachtest: perturbation/full/splits/asleep=true failed

Description

Contributor guide

First steps

Research direction

Assessment