cockroachdb / cockroachdb/cockroach

basalt-26.3.1-rc: roachtest: cluster_creation failed

Open
#173,893 7 comments 0 reactions 0 assignees View on GitHub
branch-basalt-26.3.1-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21484507?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21484507?buildTab=artifacts#/kv0/enc=false/nodes=3/cpu=32/size=64kb/splt=100) on basalt-26.3.1-rc @ [63bc2c6772a94c6d763293cd0c2ec06e359c0a36](https://github.com/cockroachdb/cockroach/commits/63bc2c6772a94c6d763293cd0c2ec06e359c0a36):

```
test kv0/enc=false/nodes=3/cpu=32/size=64kb/splt=100 failed: (test_runner.go:1017).func4: in provider: aws: failed to create instance teamcity-21484507-1787724187-15-n4cpu32-0004: aws create failed due to capacity for machine type m6i.xlarge in zone(s) [us-east-2c]; suggested zone(s): [us-east-2a us-east-2b]: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6i.xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-Eqep2QYiGkcFpmLV5d44c-Tlxfk= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e71591c1fe06645e --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=basalt-26-3-1-rc},{Key=arch,Value=amd64},{Key=created,Value=2026-08-26T06:22:56Z},{Key=Cluster,Value=teamcity-21484507-1787724187-15-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-21484507-1787724187-15-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=Name,Value=teamcity-21484507-1787724187-15-n4cpu32-0004},{Key=roachprod-data-disk,Value=boot}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=basalt-26-3-1-rc},{Key=arch,Value=amd64},{Key=created,Value=2026-08-26T06:22:56Z},{Key=Cluster,Value=teamcity-21484507-1787724187-15-n4cpu32},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-21484507-1787724187-15-n4cpu32},{Key=lifetime,Value=12h0m0s},{Key=Name,Value=teamcity-21484507-1787724187-15-n4cpu32-0004},{Key=roachprod-data-disk,Value=boot}] --user-data file:///tmp/aws-startup-script2540531357 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2212577764 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr:
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m6i.xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get m6i.xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254 [owner=test-eng]
```

Parameters:
- MetamorphicFilesystem=ext4
- MetamorphicVolumeType=io2
- cloud=aws
- coverageBuild=false
- cpu=32
- diskCount=0
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #173220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.15-rc]
- #173219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.23-rc]
- #173209 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.15-rc]
- #173187 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #173160 release-26.3.1-rc: roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.3.1-rc]
- #173156 release-26.2.6-rc: roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2.6-rc]
- #173155 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.15-rc]
- #173154 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.23-rc]
- #172904 basalt-26.3: roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-basalt-26.3]
- #172635 release-26.2: roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67192

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and test_runner.go around line 1017, then inspect the AWS cluster-creation path for the reported m6i.xlarge capacity failure in us-east-2c. Compare the repeated related issues and the TeamCity log to determine the intended handling; done means cluster_creation behaves as intended when AWS reports insufficient capacity.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.