aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap
Cloudformation does not wait for ECS service to stabilize
- Dominant language
- No language data
- Stars
- 1.1k
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
### Name of the resource
AWS::ECS::Service
### Resource Name
_No response_
### Issue Description
Cloudformation, when used to deploy an ECS service, does not always respect the fact that the service should reach a steady state before receiving the "CREATE COMPLETE/UPDATE_COMPLETE" status.
If an invalid image if specified in the corresponding task, the result is as expected - the service does not reach a steady state and the resource creation/update fails.
If, however, the image exists but the service does not reach a steady state because of an ECS health check, Cloudformation simply considers the resource as OK and allows it to reach "CREATE COMPLETE/UPDATE_COMPLETE" status, even before ECS gives up starting the service.
Here are other relevant issues about this:
- https://github.com/aws/containers-roadmap/issues/897
This was about the fact that ECS services reached steady state before the container healthcheck passed. This was fixed a few years ago (I've confirmed by testing today).
- https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/150
This is about the fact that Cloudformation does not recognize a failed ECS deployment, instead timing out after 3 hours. This issue is still open (and I've confirmed by testing that the problem still exists).
### Expected Behavior
Cloudformation, when used to deploy an ECS service, should wait for the service to reach a steady state before giving the resource a CREATE_COMPLETE or UPDATE_COMPLETE status.
### Observed Behavior
Cloudformation, when used to deploy an ECS service, gives the resource a CREATE_COMPLETE/UPDATE_COMPLETE status before the service reaches a steady state.
### Test Cases
- Create an ECS service with a corresponding task having a valid image and the following HealthCheck property:
HealthCheck:
Command:
- "CMD-SHELL"
- "exit 1"
Interval: 5
Retries: 1
Timeout: 2
The resource in Cloudformation will reach the CREATE_COMPLETE/UPDATE_COMPLETE status but the ECS service will not stabilize.
### Other Details
_No response_
Contributor guide
Research direction
Reproduce with an AWS::ECS::Service whose task uses a valid image and the failing HealthCheck command shown in the issue. Compare the CloudFormation CREATE_COMPLETE/UPDATE_COMPLETE status with ECS service stabilization; done means CloudFormation waits for steady state and reports failure when stabilization does not occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100