aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap

[AWS::ECS::Service] - [BUG] - EXPRESS mode does not update service when its task definition is replaced by a parameter change

Open
#2,538 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

> This report was drafted with the help of Claude, from a bug we hit and reproduced in our own account. Human reviewed though obviously.

**Resource:** `AWS::ECS::Service` · **Deployment mode:** EXPRESS (`DeploymentConfig.Mode = EXPRESS`) · **Region observed:** eu-west-1

## Issue Description

When an `AWS::ECS::TaskDefinition` is replaced because a **stack parameter** referenced by its container image changed, and an `AWS::ECS::Service` references that task definition via `{"Ref": TaskDef}`, EXPRESS mode does **not** update the service. CloudFormation registers the new task definition revision, reports `UPDATE_COMPLETE`, and even deletes the previous revision during cleanup — but emits **no `AWS::ECS::Service` event**, makes **no `ecs:UpdateService` call** (confirmed absent from CloudTrail), and leaves the service running the previous revision.

The same stack, same template, same one-parameter change, deployed in **STANDARD** mode updates the service correctly.

This is not a change-set planning problem. A change set for the update — in **either** mode — plans both resources:

```
TaskDef AWS::ECS::TaskDefinition Modify Replacement=True
Service AWS::ECS::Service Modify Replacement=False
```

So EXPRESS mode plans the service update and then does not execute it. This looks related to #2536 (EXPRESS-mode DynamoDB handler receiving the previous resource model without its full state), possibly the same underlying previous-state-propagation defect on the EXPRESS path.

## Expected Behavior

The service's `TaskDefinition` `Ref` resolves to a new revision ARN after replacement, so the service must be updated — as it is in STANDARD mode. Per the [docs](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-express-mode.html), "Express mode respects resource dependencies… Express mode does not change the dependency ordering of your resources."

## Observed Behavior

The service is stranded on the previous (now `INACTIVE`) revision; tasks keep launching the previous image indefinitely. Nothing in stack status, events, or CLI exit code signals it.

| Step | Mode | Task def | Service | `AWS::ECS::Service` event |
|------|----------|-------------|-----------------|---------------------------|
| create | EXPRESS | rev2 | rev2 | `CREATE_COMPLETE` |
| update 1 | EXPRESS | rev2 → rev3 | rev2 | none |
| update 2 | EXPRESS | rev3 → rev4 | rev2 | none |
| update 3 | EXPRESS | rev4 → rev5 | rev2 | none |
| update 4 | STANDARD | rev5 → rev6 | rev2 → **rev6** | `UPDATE_IN_PROGRESS` |

Each EXPRESS update reported `UPDATE_COMPLETE` in ~10s with no `ecs:UpdateService` in CloudTrail. The first STANDARD update cascaded and caught the service up.

## Test Cases

Self-contained; the service runs at `DesiredCount: 0`, so it needs no cluster capacity and never pulls the image — the digest only has to be a well-formed reference. Substitute any two distinct digests.

```yaml
# cascade-probe.yaml
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
ImageDigest:
Type: String
Resources:
Cluster:
Type: AWS::ECS::Cluster
Properties: { ClusterName: cfn-cascade-probe }
TaskDef:
Type: AWS::ECS::TaskDefinition
Properties:
Family: cfn-cascade-probe
NetworkMode: bridge
RequiresCompatibilities: [EC2]
ContainerDefinitions:
- Name: app
Memory: 128
Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/some-repo@${ImageDigest}"
Service:
Type: AWS::ECS::Service
Properties:
ServiceName: cfn-cascade-probe
Cluster: !Ref Cluster
TaskDefinition: !Ref TaskDef
LaunchType: EC2
DeploymentConfiguration: { MinimumHealthyPercent: 100, MaximumPercent: 200 }
```

Requires AWS CLI ≥ 2.35.13 for `--deployment-config`.

```bash
A=sha256:1111111111111111111111111111111111111111111111111111111111111111
B=sha256:2222222222222222222222222222222222222222222222222222222222222222

aws cloudformation create-stack --stack-name cfn-cascade-probe \
--template-body file://cascade-probe.yaml \
--parameters ParameterKey=ImageDigest,ParameterValue=$A \
--deployment-config '{"mode":"EXPRESS","disableRollback":false}'
aws cloudformation wait stack-create-complete --stack-name cfn-cascade-probe

# Change ONLY the parameter, in EXPRESS mode. Repeat 2-3x, alternating $B/$A.
aws cloudformation update-stack --stack-name cfn-cascade-probe \
--template-body file://cascade-probe.yaml \
--parameters ParameterKey=ImageDigest,ParameterValue=$B \
--deployment-config '{"mode":"EXPRESS","disableRollback":false}'
aws cloudformation wait stack-update-complete --stack-name cfn-cascade-probe

# Task def family advances a revision; the service does not move, and no
# AWS::ECS::Service event is emitted for any EXPRESS update:
aws ecs describe-services --cluster cfn-cascade-probe --services cfn-cascade-probe \
--query 'services[0].taskDefinition'
aws cloudformation describe-stack-events --stack-name cfn-cascade-probe \
--query 'StackEvents[?ResourceType==`AWS::ECS::Service`].[Timestamp,ResourceStatus]' --output text

# Control: the same change in STANDARD mode (omit --deployment-config) updates
# the service immediately.
aws cloudformation update-stack --stack-name cfn-cascade-probe \
--template-body file://cascade-probe.yaml \
--parameters ParameterKey=ImageDigest,ParameterValue=$A
```

Expected after the fix: every EXPRESS update that replaces the task definition also updates the service to the new revision, exactly as STANDARD mode does.

## Other Details

**Secondary issue — replacement updates rejected under the EXPRESS default.** EXPRESS disables rollback by default, and with that default *any* replacement update fails outright:

```
Replacement type updates not supported on stack with disable-rollback.
```

A task definition change is always a replacement, so default EXPRESS cannot deploy a container image change at all; passing `disableRollback: false` is what surfaces the silent-skip above instead. Recovering the failed stack additionally required `cancel-update-stack` then `continue-update-rollback --resources-to-skip`. (Compare #2536's Issue 2, where EXPRESS failed-state recovery 500s.)

**Impact.** A pipeline that feeds image digests as stack parameters — the synth-once/deploy-many pattern — silently deploys stale container images to ECS while reporting success. Where the same digest parameter also feeds Lambda functions, those *are* updated (modified in place, not replaced), so the fleet splits between updated Lambdas and stale ECS workers.

**Environment.** Reproduced with the raw CloudFormation API (botocore 1.43.50) using the template above, and originally via CDK CLI 2.1129.0 (`cdk deploy --express --rollback`). `AWS::ECS::Service` with `LaunchType: EC2`, `DesiredCount` unmanaged (absent from the template). Account/stack IDs and the production stack's RequestIds/timestamps available privately to AWS staff.

## Asks

1. Confirm whether this is a previous-state propagation defect on the EXPRESS path for the ECS service handler (the dependent resource not seeing that its `Ref`'d task definition changed).
2. A planned-but-not-executed dependent update should not report `UPDATE_COMPLETE` — the skip is silent and indistinguishable from success.

Contributor guide

Open the contributing guide

Research direction

Start by running the self-contained cascade-probe.yaml with the AWS CLI, changing ImageDigest in EXPRESS mode and comparing the result with STANDARD mode. Check describe-services, CloudFormation stack events, and CloudTrail for the missing service update. Done means each task-definition replacement updates the ECS service to the new revision and reports a corresponding service event.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.