cockroachdb / cockroachdb/cockroach

roachtest: cluster_creation failed

Open
#151,515 8 comments 0 reactions 0 assignees View on GitHub
branch-release-24.3.18-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/20308639?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/20308639?buildTab=artifacts#/kv0/enc=false/nodes=1/size=64kb/conc=4096) on release-24.3.18-rc @ [070aa784b97d56970e90734d24c15e057000cdd0](https://github.com/cockroachdb/cockroach/commits/070aa784b97d56970e90734d24c15e057000cdd0):

```
test kv0/enc=false/nodes=1/size=64kb/conc=4096 failed: (test_runner.go:829).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.2xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-73kcEzNuAO7UH3EVJ1ZyAsCtb1w= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=release-24-3-18-rc},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-20308639-1754544819-20-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2025-08-07T05:55:28Z},{Key=Name,Value=teamcity-20308639-1754544819-20-n2cpu8-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=release-24-3-18-rc},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-20308639-1754544819-20-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2025-08-07T05:55:28Z},{Key=Name,Value=teamcity-20308639-1754544819-20-n2cpu8-0001}] --user-data file:///tmp/aws-startup-script952729479 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4278533055 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr:
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]
```

Parameters:
- cloud=aws
- coverageBuild=false
- cpu=8
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #151390 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #151389 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #151388 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #151387 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #151242 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #151241 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #151240 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #151239 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #151238 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #151237 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-53321

Contributor guide

Open the contributing guide

Research direction

Start with the linked TeamCity log and artifacts, then read pkg/cmd/roachtest/README.md to understand cluster creation and AWS provider behavior. Compare the repeated cluster_creation failures listed in this issue and the referenced roachdash results; the report does not specify a code change, so completion would require identifying and addressing the recurring InsufficientInstanceCapacity failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.