aws / aws/aws-cdk

(aws-ecs / aws-elasticloadbalancingv2): CDK resets ALB listener rule actions during stack updates, breaking blue/green deployments

Open
#36,649 5 comments 5 reactions 0 assignees View on GitHub
@aws-cdk/aws-ecs bug effort/medium p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

When using ECS blue/green deployments (either with `AlternateTarget` for native blue/green or manual weighted target groups), CDK resets the listener rule action to the default state defined in code whenever ANY stack update occurs. This happens even when traffic has been shifted externally via AWS CLI, CodeDeploy, ECS native deployment, or the AWS console.

This makes it **very risky** to:
1. Perform blue/green traffic shifts outside of CDK
2. Continue to update infrastructure with CDK without disrupting live traffic routing

## Reproduction Steps

### Scenario 1: ECS Native Blue/Green with AlternateTarget

Based on the test file `packages/aws-cdk-lib/aws-ecs/test/alternate-target-configuration.test.ts`:
```ts
const prodRule = new elbv2.ApplicationListenerRule(stack, 'ProdRule', {
listener,
priority: 1,
conditions: [elbv2.ListenerCondition.pathPatterns(['/prod'])],
action: elbv2.ListenerAction.forward([blueTargetGroup]), // Hardcoded default
});

const alternateTarget = new ecs.AlternateTarget('GreenTG', {
alternateTargetGroup: greenTargetGroup,
productionListener: ecs.ListenerRuleConfiguration.applicationListenerRule(prodRule),
});

const service = new ecs.FargateService(stack, 'FargateService', {
cluster,
taskDefinition,
});

const target = service.loadBalancerTarget({
containerName: 'web',
containerPort: 80,
alternateTarget,
});
target.attachToApplicationTargetGroup(blueTargetGroup);
```

**Steps to reproduce:**
1. Deploy the stack - traffic goes to blue
2. ECS native blue/green shifts traffic to green (modifies `prodRule` action)
3. Update conditions or any other resource in Listener
4. Run `cdk deploy`
5. **Result**: The `prodRule` action is reset to `forward([blueTargetGroup])`, breaking live traffic

### Scenario 2: Weighted Target Groups
```ts
const listenerRule = new elbv2.ApplicationListenerRule(this, 'LiveTrafficRule', {
listener,
priority: 1,
conditions: [elbv2.ListenerCondition.pathPatterns(['/*'])],
action: elbv2.ListenerAction.weightedForward([
{ targetGroup: blue, weight: 100 },
{ targetGroup: green, weight: 0 }
]),
});
```

**Steps to reproduce:**
1. Deploy with traffic at blue: 100, green: 0
2. Use AWS CLI to shift traffic: `aws elbv2 modify-rule --rule-arn --actions ...` (blue: 0, green: 100)
3. Update ANY other resource in the stack
4. Run `cdk deploy`
5. **Result**: Traffic weights are reset to blue: 100, green: 0 (destroys live traffic routing)

## Expected Behavior

CDK should provide a mechanism to exclude listener rule actions from management, similar to:
- Terraform's `lifecycle { ignore_changes = [...] }`
- CloudFormation's `UpdateReplacePolicy` for specific properties

This would allow:
- CDK to manage the listener rule infrastructure (conditions, priority, target group references)
- External tools (CodeDeploy, ECS, AWS CLI) to manage traffic routing (action/weights)
- Continued CDK updates without resetting traffic state

## Current Behavior

CloudFormation always enforces the declarative state defined in CDK code, resetting any external changes to listener rule actions on every stack update.

## Possible Solution

Provide one or more of the following:

### Option 1: Property-level deletion override
```ts
const cfnRule = listenerRule.node.defaultChild as elbv2.CfnListenerRule;
cfnRule.addPropertyDeletionOverride('Actions.0.ForwardConfig.TargetGroups.0.Weight');
cfnRule.addPropertyDeletionOverride('Actions.0.ForwardConfig.TargetGroups.1.Weight');
// or for the entire action
cfnRule.addPropertyDeletionOverride('Actions.0');
```

### Option 2: Dedicated property to exclude action from management
```ts
const listenerRule = new elbv2.ApplicationListenerRule(this, 'Rule', {
listener,
conditions: [...],
action: elbv2.ListenerAction.forward([blueTargetGroup]),
manageAction: false, // Don't update the action after initial creation
});
```

### Option 3: Import pattern for external management
```ts
const listenerRule = elbv2.ApplicationListenerRule.fromListenerRuleArn(
this,
'ExistingRule',
ruleArn,
{ manageAction: false }
);
```

## Environment

- **CDK CLI Version**: 2.x
- **Framework Version**: aws-cdk-lib 2.x
- **Node.js Version**: 18.x+
- **OS**: All
- **Language**: TypeScript/JavaScript

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Library Version

_No response_

### Expected Behavior

-

### Current Behavior

-

### Reproduction Steps

-

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS CDK Library version (aws-cdk-lib)

2.233.0

### AWS CDK CLI version

2.1033.0

### Node.js Version

v24.5.0

### OS

MacOS

### Language

TypeScript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with packages/aws-cdk-lib/aws-ecs/test/alternate-target-configuration.test.ts and the ApplicationListenerRule and CfnListenerRule entry points described in the issue. Trace how listener rule Actions are synthesized and updated, then add focused coverage for externally changed actions surviving unrelated stack updates; done means the relevant blue/green and weighted-routing scenarios no longer reset traffic state.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.