aws / aws/aws-cdk

aws-ecs: Updating capacity provider strategies on EC2 service causes replacement

Open
#29,826 3 comments 2 reactions 0 assignees View on GitHub
@aws-cdk/aws-ecs bug effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

When attempting to update a service which has no capacity provider strategy (i.e. defaulted to EC2 launch type) to have a capacity provider strategy, CDK reports that it will replace the service rather than update-in-place.

This is problematic for two reasons:
1) This will result in downtime, which should not be necessary given that ECS appears to be able to update this in-place with a new deployment.
2) (Perhaps less important, or specific to my use-case) The service replacement doesn't work, and Cloudformation fails with an error that the service "already exists".``

### Expected Behavior

I expect the capacity provider strategy would be updated in-place without requiring a service replacement.

### Current Behavior

CDK/Cloudformation attempts to recreate the service (with the same name), and fails.

The output from `cdk diff`
```
[~] AWS::ECS::Service Service/Service ServiceD69D759B replace
├─ [+] CapacityProviderStrategy
│ └─ [{"Base":0,"CapacityProvider":"bkt-stage-ecs-capcity-association-ExtraLargeCapacityProvider-hMg2vMaHZjvB","Weight":1}]
├─ [~] DeploymentConfiguration
│ └─ [+] Added: .Alarms
├─ [-] LaunchType (requires replacement)
│ └─ EC2
└─ [~] DependsOn
└─ @@ -1,3 +1,5 @@
[ ] [
[+] "Ec2TaskTaskRole88E481A9",
[+] "Ec2TaskTaskRoleDefaultPolicy8BFDC019",
[ ] "Listener828B0E81"
[ ] ]
```

Fails to deploy:
```
bkt-sandbox-event-service: deploying...
[0%] start: Publishing 575056c4a80a4675f729f94d81d3c941abe89eb2e79980b1f66510651c49b041:741749084370-ap-southeast-2
[100%] success: Published 575056c4a80a4675f729f94d81d3c941abe89eb2e79980b1f66510651c49b041:741749084370-ap-southeast-2
bkt-sandbox-event-service: creating CloudFormation changeset...
bkt-sandbox-event-service | 0/6 | 1:48:57 AM | UPDATE_IN_PROGRESS | AWS::CloudFormation::Stack | bkt-sandbox-event-service User Initiated
bkt-sandbox-event-service | 0/6 | 1:49:01 AM | CREATE_IN_PROGRESS | AWS::ApiGateway::Deployment | APIGatewayDeployment (APIGatewayDeployment1E68EF6D6fcf8ac110bb22c4e72adc6284a4a82c)
bkt-sandbox-event-service | 0/6 | 1:49:04 AM | CREATE_IN_PROGRESS | AWS::ApiGateway::Deployment | APIGatewayDeployment (APIGatewayDeployment1E68EF6D6fcf8ac110bb22c4e72adc6284a4a82c) Resource creation Initiated
bkt-sandbox-event-service | 1/6 | 1:49:04 AM | CREATE_COMPLETE | AWS::ApiGateway::Deployment | APIGatewayDeployment (APIGatewayDeployment1E68EF6D6fcf8ac110bb22c4e72adc6284a4a82c)
bkt-sandbox-event-service | 1/6 | 1:49:04 AM | UPDATE_IN_PROGRESS | AWS::ECS::TaskDefinition | Ec2Task (Ec2TaskB165294F) Requested update requires the creation of a new physical resource; hence creating one.
bkt-sandbox-event-service | 1/6 | 1:49:06 AM | UPDATE_IN_PROGRESS | AWS::ECS::TaskDefinition | Ec2Task (Ec2TaskB165294F) Resource creation Initiated
bkt-sandbox-event-service | 2/6 | 1:49:06 AM | UPDATE_COMPLETE | AWS::ECS::TaskDefinition | Ec2Task (Ec2TaskB165294F)
bkt-sandbox-event-service | 2/6 | 1:49:08 AM | UPDATE_IN_PROGRESS | AWS::ECS::Service | Service/Service (ServiceD69D759B) Requested update requires the creation of a new physical resource; hence creating one.
bkt-sandbox-event-service | 2/6 | 1:49:09 AM | UPDATE_FAILED | AWS::ECS::Service | Service/Service (ServiceD69D759B) Resource handler returned message: "Resource of type 'AWS::ECS::Service' with identifier 'event-service-sandbox' already exists." (RequestToken: 37bce0e0-69a9-8a5d-865f-f222d914da26, HandlerErrorCode: AlreadyExists)
bkt-sandbox-event-service | 2/6 | 1:49:10 AM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack | bkt-sandbox-event-service The following resource(s) failed to update: [ServiceD69D759B].
bkt-sandbox-event-service | 1/6 | 1:49:13 AM | UPDATE_COMPLETE | AWS::ECS::TaskDefinition | Ec2Task (Ec2TaskB165294F)
bkt-sandbox-event-service | 2/6 | 1:49:13 AM | UPDATE_COMPLETE | AWS::ECS::Service | Service/Service (ServiceD69D759B)
bkt-sandbox-event-service | 3/6 | 1:49:15 AM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | bkt-sandbox-event-service
bkt-sandbox-event-service | 3/6 | 1:49:16 AM | DELETE_IN_PROGRESS | AWS::ApiGateway::Deployment | APIGatewayDeployment (APIGatewayDeployment1E68EF6D6fcf8ac110bb22c4e72adc6284a4a82c)
bkt-sandbox-event-service | 2/6 | 1:49:16 AM | DELETE_COMPLETE | AWS::ECS::Service | Service/Service (ServiceD69D759B)
bkt-sandbox-event-service | 2/6 | 1:49:17 AM | DELETE_IN_PROGRESS | AWS::ECS::TaskDefinition | Ec2Task (Ec2TaskB165294F)
bkt-sandbox-event-service | 1/6 | 1:49:17 AM | DELETE_COMPLETE | AWS::ApiGateway::Deployment | APIGatewayDeployment (APIGatewayDeployment1E68EF6D6fcf8ac110bb22c4e72adc6284a4a82c)
bkt-sandbox-event-service | 0/6 | 1:49:18 AM | DELETE_COMPLETE | AWS::ECS::TaskDefinition | Ec2Task (Ec2TaskB165294F)
bkt-sandbox-event-service | 1/6 | 1:49:18 AM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | bkt-sandbox-event-service
Failed resources:
bkt-sandbox-event-service | 1:49:09 AM | UPDATE_FAILED | AWS::ECS::Service | Service/Service (ServiceD69D759B) Resource handler returned message: "Resource of type 'AWS::ECS::Service' with identifier 'event-service-sandbox' already exists." (RequestToken: 37bce0e0-69a9-8a5d-865f-f222d914da26, HandlerErrorCode: AlreadyExists)
❌ bkt-sandbox-event-service failed: Error: The stack named bkt-sandbox-event-service failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Resource of type 'AWS::ECS::Service' with identifier 'event-service-sandbox' already exists." (RequestToken: 37bce0e0-69a9-8a5d-865f-f222d914da26, HandlerErrorCode: AlreadyExists)
at FullCloudFormationDeployment.monitorDeployment (/opt/atlassian/pipelines/agent/build/node_modules/aws-cdk/lib/api/deploy-stack.ts:496:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at deployStack2 (/opt/atlassian/pipelines/agent/build/node_modules/aws-cdk/lib/cdk-toolkit.ts:241:24)
at /opt/atlassian/pipelines/agent/build/node_modules/aws-cdk/lib/deploy.ts:39:11
at run (/opt/atlassian/pipelines/agent/build/node_modules/p-queue/dist/index.js:163:29)
❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named bkt-sandbox-event-service failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Resource of type 'AWS::ECS::Service' with identifier 'event-service-sandbox' already exists." (RequestToken: 37bce0e0-69a9-8a5d-865f-f222d914da26, HandlerErrorCode: AlreadyExists)
at deployStacks (/opt/atlassian/pipelines/agent/build/node_modules/aws-cdk/lib/deploy.ts:61:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at CdkToolkit.deploy (/opt/atlassian/pipelines/agent/build/node_modules/aws-cdk/lib/cdk-toolkit.ts:314:7)
at initCommandLine (/opt/atlassian/pipelines/agent/build/node_modules/aws-cdk/lib/cli.ts:357:12)
```

### Reproduction Steps

1. Create an ECS service with no capacity provider strategy.
2. Attempt to update the service to specify a capacity provider.

### Possible Solution

_No response_

### Additional Information/Context

I'm currently using this as a workaround:
```typescript
// Workaround to avoid recreating ECS service when updating capacity provider strategies
// Note: Even if we were OK with the downtime for this, it doesn't work because ECS tries to
// first create a new service which conflicts with the existing (not deleted) one.
const cfnService = service.node.defaultChild as ecs.CfnService
cfnService.capacityProviderStrategy = [
{
capacityProvider,
weight: 1,
},
]
```

### CDK CLI Version

2.44.0 (build bf32cb1)

### Framework Version

_No response_

### Node.js Version

v20.11.1

### OS

macOS Sonoma 14.2.1

### Language

TypeScript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the ECS service construct and its ecs.CfnService capacityProviderStrategy handling, comparing the generated cdk diff when changing from the default EC2 launch type. Reproduce the update and verify that adding a capacity provider strategy changes the existing service in place, without replacement, downtime, or an AlreadyExists deployment failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.