cockroachdb / cockroachdb/cockroach

basalt-26.3: roachtest: decommission/randomized failed

Open
#175,612 2 comments 0 reactions 0 assignees View on GitHub
branch-basalt-26.3 C-test-failure O-roachtest O-robot release-blocker T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.decommission/randomized [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21521860?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21521860?buildTab=artifacts#/decommission/randomized) on basalt-26.3 @ [cae870e756f1463717208bb18711995458e753cf](https://github.com/cockroachdb/cockroach/commits/cae870e756f1463717208bb18711995458e753cf):

Failed with:

```
(decommission.go:445).runDecommissionRandomized: decommission failed: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommission/randomized/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21521860-1789710892-72-n6cpu4-0001 | | 10.91.100.232 |
| teamcity-21521860-1789710892-72-n6cpu4-0002 | | 10.91.100.191 |
| teamcity-21521860-1789710892-72-n6cpu4-0003 | | 10.91.100.84 |
| teamcity-21521860-1789710892-72-n6cpu4-0004 | | 10.91.100.144 |
| teamcity-21521860-1789710892-72-n6cpu4-0005 | | 10.91.100.212 |
| teamcity-21521860-1789710892-72-n6cpu4-0006 | | 10.91.100.231 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=expiration
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21521860/decommission-randomized/1789723627598/1789723842740)

_Logs were not uploaded to Datadog_

Same failure on other branches

- #172092 release-26.3: roachtest: decommission/randomized failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.3 v26.4.0-prerelease]

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*decommission/randomized.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68499

Contributor guide

Open the contributing guide

Research direction

Start with the failure at pkg/cmd/roachtest/decommission.go:445 and inspect the artifacts and logs under /artifacts/decommission/randomized/run_1. Read the roachtest README, review the related failure in #172092, and compare the Grafana data. Done means identifying the cause of the randomized decommission failure and confirming the roachtest passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.