cockroachdb / cockroachdb/cockroach

release-26.2: roachtest: c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed

Open
#171,902 9 comments 0 reactions 1 assignee Claimed by @msbutler View on GitHub
A-disaster-recovery branch-release-26.2 C-test-failure O-roachtest O-robot P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.c2c/tpcc/warehouses=1000/duration=60/cutover=30 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21396868?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21396868?buildTab=artifacts#/c2c/tpcc/warehouses=1000/duration=60/cutover=30) on release-26.2 @ [892a1c29fbbaeff4886ff54dfbfa3a116f303aec](https://github.com/cockroachdb/cockroach/commits/892a1c29fbbaeff4886ff54dfbfa3a116f303aec):

Failed with:

```
(latency_verifier.go:211).assertValid: stream-ingestion: max latency was more than allowed: 2m0.887459079s vs 2m0s
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21396868-1782195012-48-n9cpu8-0001 | 35.231.194.145 | 10.142.1.222 |
| teamcity-21396868-1782195012-48-n9cpu8-0002 | 34.24.26.63 | 10.142.1.91 |
| teamcity-21396868-1782195012-48-n9cpu8-0003 | 35.243.185.56 | 10.142.0.210 |
| teamcity-21396868-1782195012-48-n9cpu8-0004 | 35.243.128.49 | 10.142.2.45 |
| teamcity-21396868-1782195012-48-n9cpu8-0005 | 34.24.62.208 | 10.142.2.37 |
| teamcity-21396868-1782195012-48-n9cpu8-0006 | 34.138.37.41 | 10.142.0.139 |
| teamcity-21396868-1782195012-48-n9cpu8-0007 | 34.138.8.115 | 10.142.1.196 |
| teamcity-21396868-1782195012-48-n9cpu8-0008 | 35.196.164.180 | 10.142.2.53 |
| teamcity-21396868-1782195012-48-n9cpu8-0009 | 34.74.42.107 | 10.142.1.225 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- metamorphicLeases=leader
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21396868/c2c-tpcc-warehouses-1000-duration-60-cutover-30/1782201847290/1782207203072)

/cc @cockroachdb/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*c2c/tpcc/warehouses=1000/duration=60/cutover=30.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65036

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.