cockroachdb / cockroachdb/cockroach

roachtest: cluster_creation failed

Open
#172,506 335 comments 0 reactions 0 assignees View on GitHub
branch-release-25.2.22-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21421966?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21421966?buildTab=artifacts#/admission-control/disk-bandwidth-limiter) on release-25.2.22-rc @ [2104a30430492bf810437ec7c36637226dc8be04](https://github.com/cockroachdb/cockroach/commits/2104a30430492bf810437ec7c36637226dc8be04):

```
test admission-control/disk-bandwidth-limiter failed: (test_runner.go:840).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7g.xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-GDbO2xTKbmAaZJOMO4JRTfbxlSE= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=release-25-2-22-rc},{Key=cluster,Value=teamcity-21421966-1784026091-05-n2cpu8},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2026-07-14T10:52:11Z},{Key=Name,Value=teamcity-21421966-1784026091-05-n2cpu8-0002},{Key=Cluster,Value=teamcity-21421966-1784026091-05-n2cpu8},{Key=spot,Value=true},{Key=roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=release-25-2-22-rc},{Key=cluster,Value=teamcity-21421966-1784026091-05-n2cpu8},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=created,Value=2026-07-14T10:52:11Z},{Key=Name,Value=teamcity-21421966-1784026091-05-n2cpu8-0002},{Key=Cluster,Value=teamcity-21421966-1784026091-05-n2cpu8},{Key=spot,Value=true},{Key=roachprod,Value=true}] --user-data file:///tmp/aws-startup-script2664050727 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping625591110 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr:
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7g.xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7g.xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]
```

Parameters:
- MetamorphicFilesystem=ext4
- MetamorphicVolumeType=local-ssd
- cloud=aws
- coverageBuild=false
- cpu=8
- diskCount=0
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #172191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4]
- #172020 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
- #171624 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.6-rc]
- #171623 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.12-rc]
- #171620 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.20-rc]
- #171526 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.30-rc]
- #171104 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-65754

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the failure reported at test_runner.go:840, then inspect the linked TeamCity log and artifacts for the AWS RunInstances failure. Compare the repeated failures listed for other branches and determine the appropriate cluster-creation behavior; done means the affected roachtest no longer fails because of this capacity condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, infrastructure, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.