cockroachdb / cockroachdb/cockroach

release-26.2.7-rc: roachtest: cluster_creation failed

Open
#175,511 3 comments 0 reactions 0 assignees View on GitHub
branch-release-26.2.7-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21516953?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21516953?buildTab=artifacts#/kv0/enc=false/nodes=3/cpu=96) on release-26.2.7-rc @ [d81e56ae7d745f070c6fbc5095a1eb2574616ebe](https://github.com/cockroachdb/cockroach/commits/d81e56ae7d745f070c6fbc5095a1eb2574616ebe):

```
test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:1027).func4: in provider: gce: BulkInsert failed for zone us-east1-d: BulkInsert operation failed for zone us-east1-d: gce create failed due to capacity in zone(s) [us-east1-d]: grpc: SERVICE UNAVAILABLE: errors:{code:"VM_MIN_COUNT_NOT_REACHED" message:"Requested minimum count of 3 VMs could not be created."} errors:{code:"ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS" error_details:{help:{links:{description:"Troubleshooting documentation" url:"https://cloud.google.com/compute/docs/resource-error"}}} error_details:{localized_message:{locale:"en-US" message:"A n2d-custom-96-196608 VM instance is currently unavailable in the us-east1-d zone. Consider trying your request in another zone which has capacity to accommodate your request. Alternatively, you can try your request again with a different VM hardware configuration or at a later time. For more information, see the troubleshooting documentation."}} error_details:{error_info:{domain:"compute.googleapis.com" metadatas:{key:"attachment" value:""} metadatas:{key:"vmType" value:"n2d-custom-96-196608"} metadatas:{key:"zone" value:"us-east1-d"} metadatas:{key:"zonesAvailable" value:""} reason:"stockout"}} message:"The zone 'projects/crl-e2e-infra/zones/us-east1-d' does not have enough resources available to fulfill the request. '(resource type:compute)'."} [code 2/Unknown] [owner=test-eng]
```

Parameters:
- MetamorphicFilesystem=ext4
- MetamorphicVolumeType=pd-ssd
- cloud=gce
- coverageBuild=false
- cpu=96
- diskCount=0
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68424

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the failure at test_runner.go:1027, then inspect the cluster_creation path and its GCE provider handling. Use the linked TeamCity log and artifacts to determine whether the capacity failure is reproducible and what concrete remediation is required; done should mean the test no longer fails for this capacity condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.