cockroachdb / cockroachdb/cockroach

release-26.2: roachtest: perturbation/full/backup failed

Open
#166,994 1 comment 0 reactions 0 assignees View on GitHub
A-testing branch-release-26.2 C-bug C-test-failure O-perturbation O-roachtest O-robot P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.perturbation/full/backup [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21288783?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21288783?buildTab=artifacts#/perturbation/full/backup) on release-26.2 @ [64881a0a13a908bb7791d0f831cedcc78ece9130](https://github.com/cockroachdb/cockroach/commits/64881a0a13a908bb7791d0f831cedcc78ece9130):

Failed with:

```
(assertions.go:387).Fail:
Error Trace: pkg/cmd/roachtest/tests/perturbation/framework.go:717
pkg/cmd/roachtest/test_runner.go:1559
src/runtime/asm_amd64.s:1693
Error: Should be true
Test: perturbation/full/backup
Messages: FAILURE: follower-read : Increase 76.3443 > 15.0000 BASE: 8.67498ms SCORE: 662.285467ms

FAILURE: read : Increase 91.5412 > 15.0000 BASE: 9.26677ms SCORE: 848.291265ms

FAILURE: write : Increase 70.3531 > 15.0000 BASE: 9.319527ms SCORE: 655.657207ms
(require.go:2070).True: FailNow called
test artifacts and logs in: /artifacts/perturbation/full/backup/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21288783-1774765513-288-n13cpu16sm-0001 | 104.196.27.162 | 10.142.0.139 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0002 | 35.231.13.91 | 10.142.0.142 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0003 | 34.24.177.218 | 10.142.0.135 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0004 | 35.185.126.127 | 10.142.0.154 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0005 | 35.229.23.14 | 10.142.0.137 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0006 | 34.75.78.250 | 10.142.0.161 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0007 | 34.74.121.76 | 10.142.0.140 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0008 | 34.75.255.59 | 10.142.0.158 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0009 | 34.23.232.63 | 10.142.0.152 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0010 | 35.243.155.83 | 10.142.0.155 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0011 | 35.237.108.161 | 10.142.0.147 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0012 | 104.196.213.235 | 10.142.0.143 |
| teamcity-21288783-1774765513-288-n13cpu16sm-0013 | 34.23.157.228 | 10.142.0.156 |

Parameters:
- acMode=defaultOption
- arch=amd64
- blockSize=4096
- cloud=gce
- coverageBuild=false
- cpu=16
- diskBandwidthLimit=0
- diskCount=2
- disks=2
- encrypted=false
- fillDuration=10m0s
- fs=ext4
- leaseType=leader
- localSSD=true
- mem=standard
- numNodes=12
- numWorkloadNodes=1
- perturbationDuration=10m0s
- ratioOfMax=0.5
- runtimeAssertionsBuild=false
- seed=0
- splits=10000
- validationDuration=5m0s
- vcpu=16
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21288783/perturbation-full-backup/1774823038217/1774824755603)

/cc @cockroachdb/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*perturbation/full/backup.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-62202

Contributor guide

Open the contributing guide

Research direction

Start with the linked TeamCity artifacts and logs, then read pkg/cmd/roachtest/tests/perturbation/framework.go around line 717 and the roachtest README. Compare the failed run with its Grafana data and the listed parameters. Done means the failure is explained and the equivalent perturbation/full/backup run completes without the reported latency-threshold failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
distributed-systems, performance, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.