cockroachdb / cockroachdb/cockroach

roachtest: c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed

Open
#172,875 5 comments 0 reactions 1 assignee Claimed by @stevendanna View on GitHub
A-disaster-recovery branch-release-26.1 C-bug C-test-failure O-roachtest O-robot P-2 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.c2c/tpcc/warehouses=1000/duration=60/cutover=30 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21442306?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21442306?buildTab=artifacts#/c2c/tpcc/warehouses=1000/duration=60/cutover=30) on release-26.1 @ [79cb2cd6c6120416aa085d474bc3bdfa32ce4ebc](https://github.com/cockroachdb/cockroach/commits/79cb2cd6c6120416aa085d474bc3bdfa32ce4ebc):

Failed with:

```
(latency_verifier.go:211).assertValid: stream-ingestion: max latency was more than allowed: 2m0.429664524s vs 2m0s
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/cpu_arch=fips/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21442306-1785217796-50-n9cpu8-0001 | 35.231.86.115 | 10.142.4.27 |
| teamcity-21442306-1785217796-50-n9cpu8-0002 | 34.23.209.239 | 10.142.4.28 |
| teamcity-21442306-1785217796-50-n9cpu8-0003 | 35.227.100.203 | 10.142.2.241 |
| teamcity-21442306-1785217796-50-n9cpu8-0004 | 34.23.127.4 | 10.142.2.222 |
| teamcity-21442306-1785217796-50-n9cpu8-0005 | 35.231.138.246 | 10.142.4.109 |
| teamcity-21442306-1785217796-50-n9cpu8-0006 | 34.24.185.197 | 10.142.4.30 |
| teamcity-21442306-1785217796-50-n9cpu8-0007 | 34.23.97.196 | 10.142.4.108 |
| teamcity-21442306-1785217796-50-n9cpu8-0008 | 104.196.178.122 | 10.142.4.26 |
| teamcity-21442306-1785217796-50-n9cpu8-0009 | 136.108.85.154 | 10.142.2.32 |

Parameters:
- arch=fips
- cloud=gce
- coverageBuild=false
- cpu=8
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- metamorphicLeases=default
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21442306/c2c-tpcc-warehouses-1000-duration-60-cutover-30/1785224708956/1785229355803)

/cc @cockroachlabs/disaster-recovery

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*c2c/tpcc/warehouses=1000/duration=60/cutover=30.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66146

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.