cockroachdb / cockroachdb/cockroach

roachtest: replicate/rebalance/3to5 failed

Open
#172,781 11 comments 0 reactions 1 assignee Claimed by @stevendanna View on GitHub
A-testing branch-master C-bug C-test-failure O-roachtest O-robot P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.replicate/rebalance/3to5 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21437582?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21437582?buildTab=artifacts#/replicate/rebalance/3to5) on master @ [879ec108affe17aa5317485bbb5032a512f27ff1](https://github.com/cockroachdb/cockroach/commits/879ec108affe17aa5317485bbb5032a512f27ff1):

Failed with:

```
(monitor.go:267).Wait: monitor failure: 121s elapsed without changes, but replica count standard deviation is 57.15 (>42.00)
test artifacts and logs in: /artifacts/replicate/rebalance/3to5/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21437582-1784874033-88-n6cpu4-0001 | 52.188.225.86 | 10.1.0.100 |
| teamcity-21437582-1784874033-88-n6cpu4-0002 | 20.127.185.96 | 10.1.0.154 |
| teamcity-21437582-1784874033-88-n6cpu4-0003 | 20.102.90.208 | 10.1.0.15 |
| teamcity-21437582-1784874033-88-n6cpu4-0004 | 20.51.236.10 | 10.1.0.155 |
| teamcity-21437582-1784874033-88-n6cpu4-0005 | 138.91.108.1 | 10.1.0.150 |
| teamcity-21437582-1784874033-88-n6cpu4-0006 | 20.62.252.120 | 10.1.0.10 |

Parameters:
- arch=amd64
- cloud=azure
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- metamorphicLeases=default
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21437582/replicate-rebalance-3to5/1784884521443/1784888007008)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21437582-1784874033-88-n6cpu4&from_ts=1784884521443&to_ts=1784888007008&live=false&storage=flex_tier)

Same failure on other branches

- #172761 release-26.3.0-rc: roachtest: replicate/rebalance/3to5 failed [C-test-failure O-roachtest O-robot T-kv branch-release-26.3.0-rc release-blocker]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*replicate/rebalance/3to5.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66048

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.