cockroachdb / cockroachdb/cockroach

release-26.3: roachtest: replicagc-changed-peers/restart=false failed

Open
#171,810 4 comments 0 reactions 0 assignees View on GitHub
A-testing B-runtime-assertions-enabled branch-release-26.3 C-bug C-test-failure O-roachtest O-robot P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Note:** This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.replicagc-changed-peers/restart=false [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21393548?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21393548?buildTab=artifacts#/replicagc-changed-peers/restart=false) on release-26.3 @ [769f38f9d3492a70416a81f5ed38c5f09f4b4b83](https://github.com/cockroachdb/cockroach/commits/769f38f9d3492a70416a81f5ed38c5f09f4b4b83):

Failed with:

```
(replicagc.go:96).runReplicaGCChangedPeers: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/replicagc-changed-peers/restart=false/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21393548-1781849444-104-n6cpu4-0001 | 34.24.90.237 | 10.142.1.232 |
| teamcity-21393548-1781849444-104-n6cpu4-0002 | 34.138.24.25 | 10.142.1.222 |
| teamcity-21393548-1781849444-104-n6cpu4-0003 | 35.231.39.205 | 10.142.1.240 |
| teamcity-21393548-1781849444-104-n6cpu4-0004 | 34.148.54.26 | 10.142.1.224 |
| teamcity-21393548-1781849444-104-n6cpu4-0005 | 34.73.149.62 | 10.142.1.237 |
| teamcity-21393548-1781849444-104-n6cpu4-0006 | 35.243.198.131 | 10.142.0.108 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- metamorphicLeases=expiration
- runtimeAssertionsBuild=true
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21393548/replicagc-changed-peers-restart-false/1781883696163/1781883921290)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21393548-1781849444-104-n6cpu4&from_ts=1781883696163&to_ts=1781883921290&live=false&storage=flex_tier)

/cc @cockroachdb/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*replicagc-changed-peers/restart=false.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-64946

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/replicagc.go at line 96 and inspect the failed run artifacts and logs under /artifacts/replicagc-changed-peers/restart=false/run_1. Read the roachtest README and compare the runtime-assertions failure with the linked TeamCity and observability reports. Done means identifying the cause of the failed test and providing a verified fix or a clearly documented test/report change.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.