cockroachdb / cockroachdb/cockroach
basalt-26.3: roachtest: decommission/randomized failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
roachtest.decommission/randomized [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21521860?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21521860?buildTab=artifacts#/decommission/randomized) on basalt-26.3 @ [cae870e756f1463717208bb18711995458e753cf](https://github.com/cockroachdb/cockroach/commits/cae870e756f1463717208bb18711995458e753cf):
Failed with:
```
(decommission.go:445).runDecommissionRandomized: decommission failed: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommission/randomized/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21521860-1789710892-72-n6cpu4-0001 | | 10.91.100.232 |
| teamcity-21521860-1789710892-72-n6cpu4-0002 | | 10.91.100.191 |
| teamcity-21521860-1789710892-72-n6cpu4-0003 | | 10.91.100.84 |
| teamcity-21521860-1789710892-72-n6cpu4-0004 | | 10.91.100.144 |
| teamcity-21521860-1789710892-72-n6cpu4-0005 | | 10.91.100.212 |
| teamcity-21521860-1789710892-72-n6cpu4-0006 | | 10.91.100.231 |
Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=expiration
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21521860/decommission-randomized/1789723627598/1789723842740)
_Logs were not uploaded to Datadog_
Same failure on other branches
- #172092 release-26.3: roachtest: decommission/randomized failed [A-testing B-runtime-assertions-enabled C-bug C-test-failure O-roachtest O-robot P-3 T-kv branch-release-26.3 v26.4.0-prerelease]
/cc @cockroachlabs/kv-triage
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*decommission/randomized.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-68499
Contributor guide
Research direction
Start with the failure at pkg/cmd/roachtest/decommission.go:445 and inspect the artifacts and logs under /artifacts/decommission/randomized/run_1. Read the roachtest README, review the related failure in #172092, and compare the Grafana data. Done means identifying the cause of the randomized decommission failure and confirming the roachtest passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100