cockroachdb / cockroachdb/cockroach
release-26.2: roachtest: perturbation/full/backup failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.perturbation/full/backup [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21288783?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21288783?buildTab=artifacts#/perturbation/full/backup) on release-26.2 @ [64881a0a13a908bb7791d0f831cedcc78ece9130](https://github.com/cockroachdb/cockroach/commits/64881a0a13a908bb7791d0f831cedcc78ece9130):
Failed with:
```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/perturbation/framework.go:717
pkg/cmd/roachtest/test_runner.go:1559
src/runtime/asm_amd64.s:1693
Error: Should be true
Test: perturbation/full/backup
Messages: FAILURE: follower-read : Increase 76.3443 > 15.0000 BASE: 8.67498ms SCORE: 662.285467ms
FAILURE: read : Increase 91.5412 > 15.0000 BASE: 9.26677ms SCORE: 848.291265ms
FAILURE: write : Increase 70.3531 > 15.0000 BASE: 9.319527ms SCORE: 655.657207ms
(require.go:2070).True: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/backup/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21288783-1774765513-288-n13cpu16sm-0001 | 104.196.27.162 | 10.142.0.139 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0002 | 35.231.13.91 | 10.142.0.142 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0003 | 34.24.177.218 | 10.142.0.135 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0004 | 35.185.126.127 | 10.142.0.154 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0005 | 35.229.23.14 | 10.142.0.137 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0006 | 34.75.78.250 | 10.142.0.161 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0007 | 34.74.121.76 | 10.142.0.140 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0008 | 34.75.255.59 | 10.142.0.158 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0009 | 34.23.232.63 | 10.142.0.152 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0010 | 35.243.155.83 | 10.142.0.155 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0011 | 35.237.108.161 | 10.142.0.147 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0012 | 104.196.213.235 | 10.142.0.143 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0013 | 34.23.157.228 | 10.142.0.156 |
Parameters:
- acMode=defaultOption
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=16
- diskBandwidthLimit=0
- diskCount=2
- disks=2
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=leader
- localSSD=true
- mem=standard
- numNodes=12
- numWorkloadNodes=1
- perturbationDuration=10m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=0
- splits=10000
- validationDuration=5m0s
- vcpu=16
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21288783/perturbation-full-backup/1774823038217/1774824755603)
/cc @cockroachdb/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/full/backup.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-62202
Contributor guide
Research direction
Start with the linked TeamCity artifacts and logs, then read pkg/cmd/roachtest/tests/perturbation/framework.go around line 717 and the roachtest README. Compare the failed run with its Grafana data and the listed parameters. Done means the failure is explained and the equivalent perturbation/full/backup run completes without the reported latency-threshold failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- distributed-systems, performance, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100