cockroachdb / cockroachdb/cockroach
roachprod: GC of roachprod clusters not cleaning up all child resources for AWS managed clusters.
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
GC of roachprod clusters didnt cleanup AWS scaling groups, launch templates. Instances were deleted though from the ASG group.
Subsequent create of a roachprod cluster with the same name failed due to already exisitng resources with same name conflicts.
```
`roachprod create $USER-test --clouds aws --aws-zones us-east-2a --nodes 4 --aws-managed
2026/03/06 05:29:40 roachprod.go:123: WARN: using `awsreservedssocrlshared6994728e87801def' as username instead of `AWSReservedSSO_CRLShared_6994728e87801def'
2026/03/06 05:29:40 roachprod.go:123: WARN: using `chanderprabhjain' as username instead of `chanderprabh.jain'
2026/03/06 05:29:46 roachprod.go:1796: Creating cluster chanderprabhjain-test with 4 nodes...
2026/03/06 05:29:51 aws.go:719: imported /Users/chanderprabhjain/.ssh/google_compute_engine.pub as AWSReservedSSO_CRLShared_6994728e87801def-R91eCAvR-Epsxbgr0xf7qNMMyeQ= in region us-east-2
2026/03/06 05:29:51 aws.go:3458: Creating launch template chanderprabhjain-test-lt in region us-east-2
2026/03/06 05:29:53 support.go:214: exit status 254
2026/03/06 05:29:53 roachprod.go:1756: Cleaning up partially-created cluster (prev err: in provider: aws: failed to create launch template in region us-east-2: failed to create launch template: failed to run: aws ec2 create-launch-template --launch-template-name chanderprabhjain-test-lt --launch-template-data file:///var/folders/jg/1bz7x7j917dclwbnvfv6gbsc0000gp/T/aws-launch-template-data2526090955 --tag-specifications ResourceType=launch-template,Tags=[{Key=cluster,Value=chanderprabhjain-test},{Key=roachprod,Value=true},{Key=managed,Value=true}] --region us-east-2 --output json --profile CRLShared-cpjain: stderr:
An error occurred (InvalidLaunchTemplateName.AlreadyExistsException) when calling the CreateLaunchTemplate operation: Launch template name already in use.: exit status 254)
2026/03/06 05:30:01 roachprod.go:1760: Cleaning up OK
Error: in provider: aws: failed to create launch template in region us-east-2: failed to create launch template: failed to run: aws ec2 create-launch-template --launch-template-name chanderprabhjain-test-lt --launch-template-data file:///var/folders/jg/1bz7x7j917dclwbnvfv6gbsc0000gp/T/aws-launch-template-data2526090955 --tag-specifications ResourceType=launch-template,Tags=[{Key=cluster,Value=chanderprabhjain-test},{Key=roachprod,Value=true},{Key=managed,Value=true}] --region us-east-2 --output json --profile CRLShared-cpjain: stderr: : exit status 254
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
| pkg/roachprod/vm/vm.go:762
| github.com/cockroachdb/cockroach/pkg/roachprod/vm.ProvidersParallel.func1
| pkg/roachprod/vm/vm.go:772
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
Wraps: (2) in provider: aws
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).createManagedCluster.func1
| pkg/roachprod/vm/aws/aws.go:1170
| github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).createManagedCluster.Group.GoCtx.func3
| pkg/util/ctxgroup/ctxgroup.go:200
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
Wraps: (4) failed to create launch template in region us-east-2
Wraps: (5) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).createLaunchTemplate
| pkg/roachprod/vm/aws/aws.go:3471
| [...repeated from below...]
Wraps: (6) failed to create launch template
Wraps: (7) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommandWithContext
| pkg/roachprod/vm/aws/support.go:216
| github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommandWithContext
| pkg/roachprod/vm/aws/support.go:233
| github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).createLaunchTemplate
| pkg/roachprod/vm/aws/aws.go:3470
| github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).createManagedCluster.func1
| pkg/roachprod/vm/aws/aws.go:1168
| github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).createManagedCluster.Group.GoCtx.func3
| pkg/util/ctxgroup/ctxgroup.go:200
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_arm64.s:1268
Wraps: (8) failed to run: aws ec2 create-launch-template --launch-template-name chanderprabhjain-test-lt --launch-template-data file:///var/folders/jg/1bz7x7j917dclwbnvfv6gbsc0000gp/T/aws-launch-template-data2526090955 --tag-specifications ResourceType=launch-template,Tags=[{Key=cluster,Value=chanderprabhjain-test},{Key=roachprod,Value=true},{Key=managed,Value=true}] --region us-east-2 --output json --profile CRLShared-cpjain: stderr:
| An error occurred (InvalidLaunchTemplateName.AlreadyExistsException) when calling the CreateLaunchTemplate operation: Launch template name already in use.
Wraps: (9) exit status 254
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.withPrefix (9) *exec.ExitError
chanderprabhjain@CPs-Work-Laptop cockroach % roachprod destroy $USER-test
Error: UNCLASSIFIED_PROBLEM: unable to destroy cluster chanderprabhjain-test: cluster chanderprabhjain-test does not exist
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod.Destroy.func1
| pkg/roachprod/roachprod.go:1590
| github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.func1
| pkg/util/ctxgroup/ctxgroup.go:209
| github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GroupWorkers.Group.GoCtx.func2
| pkg/util/ctxgroup/ctxgroup.go:200
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_arm64.s:1268
Wraps: (3) unable to destroy cluster chanderprabhjain-test
Wraps: (4) cluster chanderprabhjain-test does not exist
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.withPrefix (4) *errors.errorString
`
```
Jira issue: CRDB-61100
Contributor guide
Assessment
This issue has not been assessed yet.