cockroachdb / cockroachdb/cockroach

release-26.2: roachtest: cluster_creation failed

Open
#174,920 175 comments 0 reactions 0 assignees View on GitHub
branch-release-26.2 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21506256?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21506256?buildTab=artifacts#/failover/partial/lease-gateway/lease=expiration) on release-26.2 @ [0ee8179c675a375ea859def1b5f7c0bf85943a57](https://github.com/cockroachdb/cockroach/commits/0ee8179c675a375ea859def1b5f7c0bf85943a57):

```
test failover/partial/lease-gateway/lease=expiration failed: (test_runner.go:1027).func4: not all nodes booted successfully: timed out after 5m: COMMAND_PROBLEM: exit status 1 [owner=test-eng]
```

Parameters:
- cloud=gce
- coverageBuild=false
- cpu=2
- diskCount=0
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #174730 basalt-26.3: roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake basalt-red branch-basalt-26.3]
- #174643 release-26.3.2-rc: roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3.2-rc]
- #174642 release-26.2.7-rc: roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.7-rc]
- #174641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.17-rc]
- #174639 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.24-rc]
- #174597 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174596 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174595 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174594 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #174593 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68091

Contributor guide

Open the contributing guide

Research direction

Read pkg/cmd/roachtest/README.md and test_runner.go around line 1027, then inspect the linked TeamCity log and artifacts for failover/partial/lease-gateway/lease=expiration. Compare the repeated failures listed across branches and determine the cluster-creation failure's cause; done means the roachtest no longer times out or the failure is correctly handled and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
distributed-systems, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.