cockroachdb / cockroachdb/cockroach
release-26.4: roachtest: perturbation/full/splits/asleep=false failed
Open
Nobody has claimed this yet.
branch-release-26.4
C-test-failure
O-roachtest
O-robot
release-blocker
T-kv
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.perturbation/full/splits/asleep=false failed with artifacts on release-26.4 @ b4884d371d74d172ab184bc4ef0924ccb758f3c2:
Failed with:
(splits.go:243).startPerturbation: query Cockroach node ID: dial tcp 10.91.96.185:26257: connect: connection refused
test artifacts and logs in: /artifacts/perturbation/full/splits/asleep=false/run_1
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
|---|---|---|
| teamcity-21523842-1789803587-60-n13cpu16sm-0001 | 10.91.98.223 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0002 | 10.91.98.124 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0003 | 10.91.98.126 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0004 | 10.91.98.35 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0005 | 10.91.98.133 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0006 | 10.91.98.13 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0007 | 10.91.98.129 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0008 | 10.91.98.47 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0009 | 10.91.98.227 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0010 | 10.91.98.40 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0011 | 10.91.97.229 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0012 | 10.91.96.20 | |
| teamcity-21523842-1789803587-60-n13cpu16sm-0013 | 10.91.96.185 |
Parameters:
acMode=defaultOptionarch=amd64blockSize=4096cloud=gceconcurrency=500coverageBuild=falsecpu=16diskBandwidthLimit=0diskCount=2disks=2encrypted=falsefillDuration=10m0sfollowerReadPercent=50fs=ext4leaseType=leaderlocalSSD=truemem=standardnumNodes=12numWorkloadNodes=1perturbationDuration=10m0sratioOfMax=0.5readPercent=50runtimeAssertionsBuild=falseseed=0splits=500validationDuration=5m0svcpu=16workloadConnInit=
Jira issue: CRDB-68557
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The failure is reported at splits.go:243 in roachtest. Start by reading the roachtest README and the run_1 artifacts and logs, then correlate the Grafana and Datadog data for node 10.91.96.185. Done means identifying why the node refused connections and validating the fix with the perturbation/full/splits/asleep=false test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, go
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100