aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap
[AWS::ECS::Service] - [BUG] - EXPRESS mode does not update service when its task definition is replaced by a parameter change
- Dominant language
- No language data
- Stars
- 1.1k
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
> This report was drafted with the help of Claude, from a bug we hit and reproduced in our own account. Human reviewed though obviously.
**Resource:** `AWS::ECS::Service` · **Deployment mode:** EXPRESS (`DeploymentConfig.Mode = EXPRESS`) · **Region observed:** eu-west-1
## Issue Description
When an `AWS::ECS::TaskDefinition` is replaced because a **stack parameter** referenced by its container image changed, and an `AWS::ECS::Service` references that task definition via `{"Ref": TaskDef}`, EXPRESS mode does **not** update the service. CloudFormation registers the new task definition revision, reports `UPDATE_COMPLETE`, and even deletes the previous revision during cleanup — but emits **no `AWS::ECS::Service` event**, makes **no `ecs:UpdateService` call** (confirmed absent from CloudTrail), and leaves the service running the previous revision.
The same stack, same template, same one-parameter change, deployed in **STANDARD** mode updates the service correctly.
This is not a change-set planning problem. A change set for the update — in **either** mode — plans both resources:
```
TaskDef AWS::ECS::TaskDefinition Modify Replacement=True
Service AWS::ECS::Service Modify Replacement=False
```
So EXPRESS mode plans the service update and then does not execute it. This looks related to #2536 (EXPRESS-mode DynamoDB handler receiving the previous resource model without its full state), possibly the same underlying previous-state-propagation defect on the EXPRESS path.
## Expected Behavior
The service's `TaskDefinition` `Ref` resolves to a new revision ARN after replacement, so the service must be updated — as it is in STANDARD mode. Per the [docs](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-express-mode.html), "Express mode respects resource dependencies… Express mode does not change the dependency ordering of your resources."
## Observed Behavior
The service is stranded on the previous (now `INACTIVE`) revision; tasks keep launching the previous image indefinitely. Nothing in stack status, events, or CLI exit code signals it.
| Step | Mode | Task def | Service | `AWS::ECS::Service` event |
|------|----------|-------------|-----------------|---------------------------|
| create | EXPRESS | rev2 | rev2 | `CREATE_COMPLETE` |
| update 1 | EXPRESS | rev2 → rev3 | rev2 | none |
| update 2 | EXPRESS | rev3 → rev4 | rev2 | none |
| update 3 | EXPRESS | rev4 → rev5 | rev2 | none |
| update 4 | STANDARD | rev5 → rev6 | rev2 → **rev6** | `UPDATE_IN_PROGRESS` |
Each EXPRESS update reported `UPDATE_COMPLETE` in ~10s with no `ecs:UpdateService` in CloudTrail. The first STANDARD update cascaded and caught the service up.
## Test Cases
Self-contained; the service runs at `DesiredCount: 0`, so it needs no cluster capacity and never pulls the image — the digest only has to be a well-formed reference. Substitute any two distinct digests.
```yaml
# cascade-probe.yaml
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
ImageDigest:
Type: String
Resources:
Cluster:
Type: AWS::ECS::Cluster
Properties: { ClusterName: cfn-cascade-probe }
TaskDef:
Type: AWS::ECS::TaskDefinition
Properties:
Family: cfn-cascade-probe
NetworkMode: bridge
RequiresCompatibilities: [EC2]
ContainerDefinitions:
- Name: app
Memory: 128
Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/some-repo@${ImageDigest}"
Service:
Type: AWS::ECS::Service
Properties:
ServiceName: cfn-cascade-probe
Cluster: !Ref Cluster
TaskDefinition: !Ref TaskDef
LaunchType: EC2
DeploymentConfiguration: { MinimumHealthyPercent: 100, MaximumPercent: 200 }
```
Requires AWS CLI ≥ 2.35.13 for `--deployment-config`.
```bash
A=sha256:1111111111111111111111111111111111111111111111111111111111111111
B=sha256:2222222222222222222222222222222222222222222222222222222222222222
aws cloudformation create-stack --stack-name cfn-cascade-probe \
--template-body file://cascade-probe.yaml \
--parameters ParameterKey=ImageDigest,ParameterValue=$A \
--deployment-config '{"mode":"EXPRESS","disableRollback":false}'
aws cloudformation wait stack-create-complete --stack-name cfn-cascade-probe
# Change ONLY the parameter, in EXPRESS mode. Repeat 2-3x, alternating $B/$A.
aws cloudformation update-stack --stack-name cfn-cascade-probe \
--template-body file://cascade-probe.yaml \
--parameters ParameterKey=ImageDigest,ParameterValue=$B \
--deployment-config '{"mode":"EXPRESS","disableRollback":false}'
aws cloudformation wait stack-update-complete --stack-name cfn-cascade-probe
# Task def family advances a revision; the service does not move, and no
# AWS::ECS::Service event is emitted for any EXPRESS update:
aws ecs describe-services --cluster cfn-cascade-probe --services cfn-cascade-probe \
--query 'services[0].taskDefinition'
aws cloudformation describe-stack-events --stack-name cfn-cascade-probe \
--query 'StackEvents[?ResourceType==`AWS::ECS::Service`].[Timestamp,ResourceStatus]' --output text
# Control: the same change in STANDARD mode (omit --deployment-config) updates
# the service immediately.
aws cloudformation update-stack --stack-name cfn-cascade-probe \
--template-body file://cascade-probe.yaml \
--parameters ParameterKey=ImageDigest,ParameterValue=$A
```
Expected after the fix: every EXPRESS update that replaces the task definition also updates the service to the new revision, exactly as STANDARD mode does.
## Other Details
**Secondary issue — replacement updates rejected under the EXPRESS default.** EXPRESS disables rollback by default, and with that default *any* replacement update fails outright:
```
Replacement type updates not supported on stack with disable-rollback.
```
A task definition change is always a replacement, so default EXPRESS cannot deploy a container image change at all; passing `disableRollback: false` is what surfaces the silent-skip above instead. Recovering the failed stack additionally required `cancel-update-stack` then `continue-update-rollback --resources-to-skip`. (Compare #2536's Issue 2, where EXPRESS failed-state recovery 500s.)
**Impact.** A pipeline that feeds image digests as stack parameters — the synth-once/deploy-many pattern — silently deploys stale container images to ECS while reporting success. Where the same digest parameter also feeds Lambda functions, those *are* updated (modified in place, not replaced), so the fleet splits between updated Lambdas and stale ECS workers.
**Environment.** Reproduced with the raw CloudFormation API (botocore 1.43.50) using the template above, and originally via CDK CLI 2.1129.0 (`cdk deploy --express --rollback`). `AWS::ECS::Service` with `LaunchType: EC2`, `DesiredCount` unmanaged (absent from the template). Account/stack IDs and the production stack's RequestIds/timestamps available privately to AWS staff.
## Asks
1. Confirm whether this is a previous-state propagation defect on the EXPRESS path for the ECS service handler (the dependent resource not seeing that its `Ref`'d task definition changed).
2. A planned-but-not-executed dependent update should not report `UPDATE_COMPLETE` — the skip is silent and indistinguishable from success.
Contributor guide
Research direction
Start by running the self-contained cascade-probe.yaml with the AWS CLI, changing ImageDigest in EXPRESS mode and comparing the result with STANDARD mode. Check describe-services, CloudFormation stack events, and CloudTrail for the missing service update. Done means each task-definition replacement updates the ECS service to the new revision and reports a corresponding service event.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100