aws / aws/aws-cdk

(ecs): cannot migrate existing ECS service to CODE_DEPLOY deployment controller

Open
#36,012 1 comment 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-codedeploy bug p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

### Describe the bug

Cannot migrate an existing ECS service from ECS deployment controller to CODE_DEPLOY deployment controller using CDK. When attempting to add `deploymentController: { type: DeploymentControllerType.CODE_DEPLOY }` to an existing `ApplicationLoadBalancedFargateService`, the CloudFormation deployment fails with:

Resource handler returned message: "Invalid request provided: Unable to update task definition on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment."

This occurs because CDK changes the task definition format from `{ "Ref": "TaskDefLogicalId" }` to the family name string `"TaskDefFamilyName"` when adding CODE_DEPLOY controller. This format change triggers an ECS UpdateService API call that includes the task definition parameter, which violates the ECS API constraint that CODE_DEPLOY services cannot update task definitions through UpdateService.

**Root Cause**: The `EcsDeploymentGroup` construct validates that `taskDefinition` must be specified as a family name string (line 267-273 in `deployment-group.ts`), but changing from the CloudFormation Ref format to family name format triggers a service update that is rejected by the ECS API.

### Regression Issue

- [x] Select this option if this issue appears to be a regression.

### Last Known Working CDK Library Version

2.50.0

### Expected Behavior

Should be able to seamlessly migrate an existing ECS service from ECS deployment controller to CODE_DEPLOY deployment controller by:

1. Adding `deploymentController: { type: DeploymentControllerType.CODE_DEPLOY }` to the service
2. Creating the `EcsDeploymentGroup` with blue/green configuration
3. Running `cdk deploy`

The deployment should succeed without requiring service recreation.

### Current Behavior

Deployment fails with error:

Resource handler returned message: "Invalid request provided: Unable to update task definition on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment. (Service: Ecs, Status Code
: 400, Request ID: xxx)"

CloudFormation diff shows task definition format changing:
```diff
• taskDefinition: { "Ref": "TaskDefLogicalId" }
+ taskDefinition: "TaskDefFamilyName"
```
This format change triggers UpdateService API call with task definition parameter, which is not allowed for CODE_DEPLOY services per [ECS API documentation](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html).

### Reproduction Steps

### Step 1: Initial Stack (ECS Deployment Controller)

```typescript
import * as cdk from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2';

export class ReproEcsMigrationStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);

const vpc = new ec2.Vpc(this, 'Vpc', { maxAzs: 2, natGateways: 1 });
const cluster = new ecs.Cluster(this, 'Cluster', { vpc });

const taskDef = new ecs.FargateTaskDefinition(this, 'TaskDef');
taskDef.addContainer('web', {
image: ecs.ContainerImage.fromRegistry('amazon/amazon-ecs-sample'),
portMappings: [{ containerPort: 80 }],
});

const alb = new elbv2.ApplicationLoadBalancer(this, 'ALB', { vpc, internetFacing: true });
const listener = alb.addListener('Listener', { port: 80 });
const tg = new elbv2.ApplicationTargetGroup(this, 'TG', {
vpc, port: 80, targetType: elbv2.TargetType.IP,
});
listener.addTargetGroups('TGAttach', { targetGroups: [tg] });

const service = new ecs.FargateService(this, 'Service', {
cluster,
taskDefinition: taskDef,
// deploymentController defaults to ECS
});
service.attachToApplicationTargetGroup(tg);
}
}
```

Deploy: npx cdk deploy

Result: Service uses TaskDefinition: { Ref: "TaskDef54694570" }

### Step 2: Migrate to CODE_DEPLOY (Triggers Bug)

```typescript
// Change the service to use CODE_DEPLOY deployment controller
const service = new ecs.FargateService(this, 'Service', {
cluster,
taskDefinition: taskDef,
deploymentController: { type: ecs.DeploymentControllerType.CODE_DEPLOY }, // ← Add this
});
```

Deploy: npx cdk deploy

Result:
• Task definition format changes to family name: TaskDefinition: "ReproEcsMigrationStackTaskDef1F29B791"
• CloudFormation detects property change and calls UpdateService API
• **Bug:** ECS rejects UpdateService with taskDefinition parameter for CODE_DEPLOY services

### Verified Behavior

Before migration:
```yaml
TaskDefinition:
Ref: TaskDef54694570 # CloudFormation reference
```

After migration:
```yaml
TaskDefinition: ReproEcsMigrationStackTaskDef1F29B791 # Family name string
```

This format change triggers the bug.

### Additional Information/Context

- Related to open issue #25777 (CDK version upgrade causing same error)
- Related to closed issues #23564, #31620, #23370
- Current workaround requires deleting and recreating the service, which is not acceptable for production environments
- This blocks customers from adopting blue/green deployments for existing services

### AWS CDK Library version (aws-cdk-lib)

2.171.0 (also affects 2.50.0 through latest)

### AWS CDK CLI version

2.1029.4 (build 09c0061)

### Node.js Version

v24.9.0

### OS

mac

### Language

TypeScript

### Language Version

_No response_

### Other information

**CloudFormation Template Diff:**
```json
// Before (ECS controller):
"TaskDefinition": {
"Ref": "ServiceTaskDefinition55FA0F15"
}

// After (CODE_DEPLOY controller):
"TaskDefinition": "ServiceTaskDefinitionFamily"
```

**CloudTrail Evidence:**
UpdateService API call includes both `deploymentController` and `taskDefinition` parameters, causing the rejection.

**Workarounds Attempted (all failed):**
1. Using L1 construct override: `cfnService.deploymentController = { type: 'CODE_DEPLOY' }` - Still changes format
2. Explicitly setting family name: `cfnService.taskDefinition = taskDefinition.family` - Causes EcsDeploymentGroup validation error
3. Two-step deployment - First step already fails

**Links:**
- ECS API Constraint: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html
- Related PR #22295: https://github.com/aws/aws-cdk/pull/22295
- Related Issue #25777: https://github.com/aws/aws-cdk/issues/25777

Contributor guide

Open the contributing guide

Research direction

Start by reading deployment-group.ts, especially the taskDefinition validation at lines 267-273, then reproduce the migration with the TypeScript stack and npx cdk deploy. Compare the CloudFormation diff and CloudTrail UpdateService parameters before and after adding CODE_DEPLOY. Done means an existing ECS service can migrate without a rejected UpdateService call containing taskDefinition or requiring recreation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.