cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: tpcc/multiregion/survive=zone/chaos=true failed

Open
#174,618 5 comments 0 reactions 1 assignee Assigned to @himanshu-shrivastava-crl View on GitHub
branch-release-26.3 C-test-failure O-roachtest O-robot P-3 T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.tpcc/multiregion/survive=zone/chaos=true [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21495766?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21495766?buildTab=artifacts#/tpcc/multiregion/survive=zone/chaos=true) on release-26.3 @ [703f5245358cf18917f2f0ea8be3048ec220d7aa](https://github.com/cockroachdb/cockroach/commits/703f5245358cf18917f2f0ea8be3048ec220d7aa):

Failed with:

```
(tpcc.go:581).runTPCC: error detected during DRT: error at from 2026-09-02T16:37:57Z, to 2026-09-02T16:41:39Z on metric workload_tpcc_newOrder_error_total{instance="10.91.99.103:2110"}: expected <=1000 errors, found from 11159006.000000, to 11160512.000000
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=zone/chaos=true/cpu_arch=arm64/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0001 | | 10.91.96.228 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0002 | | 10.91.96.135 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0003 | | 10.91.96.43 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0004 | | 10.91.128.114 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0005 | | 10.91.128.115 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0006 | | 10.91.128.113 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0007 | | 10.91.48.13 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0008 | | 10.91.48.15 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0009 | | 10.91.48.66 |
| teamcity-21495766-1788328822-206-n10cpu4-geo-0010 | | 10.91.99.103 |

Parameters:
- arch=arm64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=true
- fs=ext4
- localSSD=true
- metamorphicLeases=expiration
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21495766/tpcc-multiregion-survive-zone-chaos-true/1788361710231/1788368447656)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21495766-1788328822-206-n10cpu4-geo&from_ts=1788361710231&to_ts=1788368447656&live=false&storage=flex_tier)

Same failure on other branches

- #171392 roachtest: tpcc/multiregion/survive=zone/chaos=true failed [A-kv-replication B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3.26-rc]

/cc @cockroachlabs/sql-foundations

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*tpcc/multiregion/survive=zone/chaos=true.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67819

Contributor guide

Open the contributing guide

Research direction

Begin with the TeamCity build log and artifacts linked in the report, then inspect the failure at tpcc.go:581 and the roachtest README. Trace why the multiregion chaos run exceeds the new-order error threshold, and verify the test completes without the reported DRT failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.