aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap

Cloudformation does not wait for ECS service to stabilize

Open
#1,391 3 comments 5 reactions 0 assignees View on GitHub
bug NeedTriage
Dominant language
No language data
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

### Name of the resource

AWS::ECS::Service

### Resource Name

_No response_

### Issue Description

Cloudformation, when used to deploy an ECS service, does not always respect the fact that the service should reach a steady state before receiving the "CREATE COMPLETE/UPDATE_COMPLETE" status.

If an invalid image if specified in the corresponding task, the result is as expected - the service does not reach a steady state and the resource creation/update fails.

If, however, the image exists but the service does not reach a steady state because of an ECS health check, Cloudformation simply considers the resource as OK and allows it to reach "CREATE COMPLETE/UPDATE_COMPLETE" status, even before ECS gives up starting the service.

Here are other relevant issues about this:

- https://github.com/aws/containers-roadmap/issues/897
This was about the fact that ECS services reached steady state before the container healthcheck passed. This was fixed a few years ago (I've confirmed by testing today).

- https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/150
This is about the fact that Cloudformation does not recognize a failed ECS deployment, instead timing out after 3 hours. This issue is still open (and I've confirmed by testing that the problem still exists).

### Expected Behavior

Cloudformation, when used to deploy an ECS service, should wait for the service to reach a steady state before giving the resource a CREATE_COMPLETE or UPDATE_COMPLETE status.

### Observed Behavior

Cloudformation, when used to deploy an ECS service, gives the resource a CREATE_COMPLETE/UPDATE_COMPLETE status before the service reaches a steady state.

### Test Cases

- Create an ECS service with a corresponding task having a valid image and the following HealthCheck property:
HealthCheck:
Command:
- "CMD-SHELL"
- "exit 1"
Interval: 5
Retries: 1
Timeout: 2

The resource in Cloudformation will reach the CREATE_COMPLETE/UPDATE_COMPLETE status but the ECS service will not stabilize.

### Other Details

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce with an AWS::ECS::Service whose task uses a valid image and the failing HealthCheck command shown in the issue. Compare the CloudFormation CREATE_COMPLETE/UPDATE_COMPLETE status with ECS service stabilization; done means CloudFormation waits for steady state and reports failure when stabilization does not occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.