cockroachdb / cockroachdb/cockroach

roachtest: cluster_creation failed

Open
#159,237 2059 comments 0 reactions 0 assignees View on GitHub
branch-release-26.1 O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/20847153?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/20847153?buildTab=artifacts#/cdc/frontier-persistence-benchmark/interval=10m/tables=10/ranges=1000) on release-26.1 @ [e15b1ecc444676ba53266770e0b758de9ae72d22](https://github.com/cockroachdb/cockroach/commits/e15b1ecc444676ba53266770e0b758de9ae72d22):

```
test cdc/frontier-persistence-benchmark/interval=10m/tables=10/ranges=1000 failed: (test_runner.go:990).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.4xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-QwczjNpONpmi3wvdfBJYQ-C6ON4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=release-26-1},{Key=created,Value=2025-12-11T07:44:24Z},{Key=Name,Value=teamcity-20847153-1765437052-31-n4cpu16-0003},{Key=Cluster,Value=teamcity-20847153-1765437052-31-n4cpu16},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-20847153-1765437052-31-n4cpu16},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=release-26-1},{Key=created,Value=2025-12-11T07:44:24Z},{Key=Name,Value=teamcity-20847153-1765437052-31-n4cpu16-0003},{Key=Cluster,Value=teamcity-20847153-1765437052-31-n4cpu16},{Key=spot,Value=true},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-20847153-1765437052-31-n4cpu16},{Key=lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script2599156857 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping246025332 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr:
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m6id.4xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m6id.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]
```

Parameters:
- cloud=aws
- coverageBuild=false
- cpu=16
- runtimeAssertionsBuild=false
- ssd=0
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #159040 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #159039 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #159038 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #159032 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.3]
- #159031 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2]
- #158857 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #158856 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #158855 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #158854 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]
- #158853 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-57743

Contributor guide

Open the contributing guide

Research direction

Start with the roachtest README and the cdc/frontier-persistence-benchmark failure report, then inspect the cluster-creation entry point that issues the shown AWS run-instances command. Compare the repeated failures across the listed branches and determine the expected handling for unavailable us-east-2a capacity; done means cluster_creation is not reported as failed solely for this capacity error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.