cockroachdb / cockroachdb/cockroach
roachtest: perturbation/full/backup failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.perturbation/full/backup [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21440545?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21440545?buildTab=artifacts#/perturbation/full/backup) on release-25.2 @ [56bfdbf25a69f139b9180e367ab570d15c8c1ae9](https://github.com/cockroachdb/cockroach/commits/56bfdbf25a69f139b9180e367ab570d15c8c1ae9):
```
(assertions.go:369).Fail:
Error Trace: pkg/cmd/roachtest/tests/perturbation/framework.go:682
pkg/cmd/roachtest/test_runner.go:1339
src/runtime/asm_amd64.s:1700
Error: Should be true
Test: perturbation/full/backup
Messages: FAILURE: follower-read : Increase 5.4593 > 5.0000 BASE: 4.905587ms SCORE: 26.78101ms
FAILURE: read : Increase 5.4622 > 5.0000 BASE: 4.809628ms SCORE: 26.271273ms
(require.go:2014).True: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/backup/run_1
```
Parameters:
- acMode=defaultOption
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=16
- diskBandwidthLimit=0
- diskCount=2
- disks=2
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=epoch
- localSSD=true
- mem=standard
- numNodes=12
- numWorkloadNodes=1
- perturbationDuration=10m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=0
- splits=10000
- validationDuration=5m0s
- vcpu=16
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21440545/perturbation-full-backup/1785152174922/1785153883317)
Same failure on other branches
- #172489 roachtest: perturbation/full/backup failed [A-kv C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.1]
- #171157 roachtest: perturbation/full/backup failed [C-test-failure O-perturbation O-roachtest O-robot P-3 T-kv X-duplicate branch-release-25.4.11-rc]
- #170480 roachtest: perturbation/full/backup failed [A-testing C-bug C-test-failure O-perturbation O-roachtest O-robot P-3 T-kv X-duplicate branch-release-25.4]
/cc @cockroachlabs/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/full/backup.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-66112
Contributor guide
Research direction
Start at pkg/cmd/roachtest/tests/perturbation/framework.go:682 and inspect the perturbation/full/backup test. Review the linked TeamCity logs, artifacts, and Grafana run, then compare the related failures in issues #172489, #171157, and #170480. Done means identifying the cause of the follower-read and read threshold failures and restoring a passing test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100