cockroachdb / cockroachdb/cockroach
release-26.4: roachtest: perturbation/full/restart failed
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.perturbation/full/restart failed with artifacts on release-26.4 @ b4884d371d74d172ab184bc4ef0924ccb758f3c2:
Failed with:
(assertions.go:388).Fail:
Error Trace: pkg/cmd/roachtest/tests/perturbation/framework.go:1084
pkg/cmd/roachtest/test_runner.go:1992
src/runtime/asm_amd64.s:1771
Error: FAILURE: follower-read : throughput ratio 6508200000000.00x exceeds limit 1.2x (baseline: 6508 ops/s, measured: 0 ops/s)
FAILURE: read : throughput ratio 6515200000000.00x exceeds limit 1.2x (baseline: 6515 ops/s, measured: 0 ops/s)
FAILURE: write : throughput ratio 13090900000000.00x exceeds limit 1.2x (baseline: 13091 ops/s, measured: 0 ops/s)
FAILURE: follower-read : throughput ratio 1.40x exceeds limit 1.2x (baseline: 6508 ops/s, measured: 4640 ops/s)
FAILURE: read : throughput ratio 1.39x exceeds limit 1.2x (baseline: 6515 ops/s, measured: 4695 ops/s)
FAILURE: write : throughput ratio 1.39x exceeds limit 1.2x (baseline: 13091 ops/s, measured: 9397 ops/s)
Test: perturbation/full/restart
(require.go:522).Fail: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/restart/run_1
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
|---|---|---|
| teamcity-21523842-1789803587-06-n13cpu16sm-0001 | 10.91.101.113 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0002 | 10.91.101.101 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0003 | 10.91.101.136 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0004 | 10.91.101.137 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0005 | 10.91.101.115 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0006 | 10.91.101.119 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0007 | 10.91.101.114 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0008 | 10.91.101.100 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0009 | 10.91.101.109 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0010 | 10.91.101.132 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0011 | 10.91.101.71 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0012 | 10.91.100.92 | |
| teamcity-21523842-1789803587-06-n13cpu16sm-0013 | 10.91.101.73 |
Parameters:
acMode=defaultOptionarch=amd64blockSize=4096cloud=gceconcurrency=500coverageBuild=falsecpu=16diskBandwidthLimit=0diskCount=2disks=2encrypted=falsefillDuration=10m0sfollowerReadPercent=50fs=ext4leaseType=leaderlocalSSD=truemem=standardnumNodes=12numWorkloadNodes=1perturbationDuration=10m0sratioOfMax=0.3readPercent=50runtimeAssertionsBuild=falseseed=0splits=10000validationDuration=5m0svcpu=16workloadConnInit=
Jira issue: CRDB-68546
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the failure at pkg/cmd/roachtest/tests/perturbation/framework.go:1084 and the caller in pkg/cmd/roachtest/test_runner.go:1992. Inspect the run_1 artifacts and linked Grafana and Datadog data to determine why throughput falls to zero or below the 1.2x limit during perturbation/full/restart. Done means the failure cause is addressed and this roachtest no longer reports the shown throughput-ratio failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100