cockroachdb / cockroachdb/cockroach

roachtest: cluster_creation failed

Open
#166,254 479 comments 0 reactions 0 assignees View on GitHub
branch-master O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.cluster_creation [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21275035?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAwsBazel/21275035?buildTab=artifacts#/kv95/enc=false/nodes=1/cpu=32/mt-shared-process) on master @ [b6ba26ae45cfe48aa668e8deeadd0519a72381fe](https://github.com/cockroachdb/cockroach/commits/b6ba26ae45cfe48aa668e8deeadd0519a72381fe):

```
test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:998).func4: in provider: aws: failed to create instance teamcity-21275035-1773987908-44-n2cpu32-0001: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-ESjWpjT7ZSzeP2FZXtqOpSx_un4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=master},{Key=arch,Value=arm64},{Key=created,Value=2026-03-20T07:14:29Z},{Key=Name,Value=teamcity-21275035-1773987908-44-n2cpu32-0001},{Key=Cluster,Value=teamcity-21275035-1773987908-44-n2cpu32},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-21275035-1773987908-44-n2cpu32},{Key=lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=roachtest-branch,Value=master},{Key=arch,Value=arm64},{Key=created,Value=2026-03-20T07:14:29Z},{Key=Name,Value=teamcity-21275035-1773987908-44-n2cpu32-0001},{Key=Cluster,Value=teamcity-21275035-1773987908-44-n2cpu32},{Key=roachprod,Value=true},{Key=cluster,Value=teamcity-21275035-1773987908-44-n2cpu32},{Key=lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script947517224 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1382850736 --output json: stderr:
An error occurred (MaxIOPSLimitExceeded) when calling the RunInstances operation: You have exceeded your maximum io2 IOPS limit of 100000 IOPS in this region. Please contact AWS Support to request an Elastic Block Store service limit increase.: exit status 254 [owner=test-eng]
```

Parameters:
- MetamorphicFilesystem=xfs
- MetamorphicVolumeType=io2
- cloud=aws
- coverageBuild=false
- cpu=32
- diskCount=0
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #166139 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1.2-rc]
- #166137 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.4.7-rc]
- #166136 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-25.2.16-rc]
- #166135 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.30-rc]
- #165927 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1]
- #165926 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1]
- #165925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.1]
- #165924 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-26.1]
- #165923 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-26.1]

/cc @cockroachdb/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*cluster_creation.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-61815

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/README.md, then inspect the linked TeamCity log and artifacts for the cluster_creation failure. Investigate the AWS MaxIOPSLimitExceeded error for the io2 configuration and related repeated failures; done means cluster creation no longer fails for this limit condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.