cockroachdb / cockroachdb/cockroach

basalt-26.3: roachtest: cluster_creation failed

Open
#174,394 343 comments 0 reactions 0 assignees View on GitHub
basalt-green branch-basalt-26.3 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21491965?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_GreenRoachtestNightlyGcePrivateBasaltBazel/21491965?buildTab=artifacts#/acceptance/event-log) on basalt-26.3 @ [ac8dd0a5e6674df51e4d0f0f292029b14dbc084c](https://github.com/cockroachdb/cockroach/commits/ac8dd0a5e6674df51e4d0f0f292029b14dbc084c):

```
test acceptance/event-log failed: (test_runner.go:1027).func4: creating Basalt backing cluster teamcity-21491965-1788155646-20-n4cpu4-basalt: in provider: gce: BulkInsert failed for zone us-east1-d: BulkInsert operation failed for zone us-east1-d: gce create failed due to capacity in zone(s) [us-east1-d]: grpc: SERVICE UNAVAILABLE: errors:{code:"VM_MIN_COUNT_NOT_REACHED" message:"Requested minimum count of 3 VMs could not be created."} errors:{code:"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS" error_details:{help:{links:{description:"Troubleshooting documentation" url:"https://cloud.google.com/compute/docs/resource-error"}}} error_details:{localized_message:{locale:"en-US" message:"A c4a-standard-4-lssd VM instance with 1 local SSD(s) is currently unavailable in the us-east1-d zone. Consider trying your request in the us-east1-b zone(s), which currently has capacity to accommodate your request. Alternatively, you can try your request again with a different VM hardware configuration or at a later time. For more information, see the troubleshooting documentation."}} error_details:{error_info:{domain:"compute.googleapis.com" metadatas:{key:"attachment" value:"local-ssd:1"} metadatas:{key:"vmType" value:"c4a-standard-4-lssd"} metadatas:{key:"zone" value:"us-east1-d"} metadatas:{key:"zonesAvailable" value:"us-east1-b"} reason:"resource_availability"}} message:"The zone 'projects/crl-e2e-infra/zones/us-east1-d' does not have enough resources available to fulfill the request. 'NULL:0/NULL:0/NULL:0 (state:STOCKOUT, sub-state:STOCKOUT, resource type:compute)'."} [code 2/Unknown] [owner=test-eng]
```

Parameters:
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Contributor guide

Open the contributing guide

Research direction

Read pkg/cmd/roachtest/README.md and inspect test_runner.go around line 1027. Review the linked TeamCity failure and artifacts to understand the cluster_creation failure in GCE zone us-east1-d. Done means establishing a reliable resolution for this failure and verifying that the roachtest completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.