[ECS] [Deployment Circuit Breaker]: Cloudformation events shows ECS service as CREATE_COMPLETE when deployment circuit breaker has marked the ECS service deployment as failed.
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
What do you want us to build?
ECS Fargate task using CloudFormation.
**Which service(s) is this request for?**
This could be Fargate, ECS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.
Created an ECS Service through CloudFormation with invalid container health check configuration in task definition. I noticed that tasks were failing with container health checks and are stopped. Then, circuit breaker marked the deployment as failed after the failure threshold of stopped tasks has been met.
However, CloudFormation marked as the ECS service as CREATE_COMPLETE. I expect the AWS::ECS::Service resource to timeout and CloudFormation should mark it as CREATE_FAILED.
CloudFormation Events:
```
2021-05-10 16:18:12 UTC-0400 | CircuitBreaker | CREATE_COMPLETE | -
-- | -- | -- | --
2021-05-10 16:18:10 UTC-0400 | service | CREATE_COMPLETE | -
2021-05-10 16:17:36 UTC-0400 | service | CREATE_IN_PROGRESS | Resource creation Initiated
2021-05-10 16:17:34 UTC-0400 | service | CREATE_IN_PROGRESS | -
```
ECS Service Events:
```
2021-05-10 16:26:36 -0400 service CircuitBreakerTest (deployment ecs-svc/5889940544547797395) deployment failed: tasks failed to start.
2021-05-10 16:26:36 -0400 service CircuitBreakerTest has stopped 1 running tasks: task 765e8d82b6fc422d96ec90dd19130117.
2021-05-10 16:26:36 -0400 service CircuitBreakerTest task 765e8d82b6fc422d96ec90dd19130117 failed container health checks.
2021-05-10 16:25:44 -0400 service CircuitBreakerTest has started 1 tasks: task 765e8d82b6fc422d96ec90dd19130117.
.
.
.
.
.
.
.
.
.
2021-05-10 16:20:02 -0400 service CircuitBreakerTest has stopped 1 running tasks: task 9eb497e4097b44b28961126d8266fd0e.
2021-05-10 16:20:02 -0400 service CircuitBreakerTest task 9eb497e4097b44b28961126d8266fd0e failed container health checks.
2021-05-10 16:19:30 -0400 service CircuitBreakerTest has started 1 tasks: task 9eb497e4097b44b28961126d8266fd0e.
2021-05-10 16:19:18 -0400 service CircuitBreakerTest has stopped 1 running tasks: task 5a807d12df914f48a1eced88ec9c82fd.
2021-05-10 16:19:18 -0400 service CircuitBreakerTest task 5a807d12df914f48a1eced88ec9c82fd failed container health checks.
2021-05-10 16:18:27 -0400 service CircuitBreakerTest has started 1 tasks: task 5a807d12df914f48a1eced88ec9c82fd.
2021-05-10 16:18:16 -0400 service CircuitBreakerTest has stopped 1 running tasks: task f35ea31f9b574bc787fe5fad80f49a4d.
2021-05-10 16:18:16 -0400 service CircuitBreakerTest task f35ea31f9b574bc787fe5fad80f49a4d failed container health checks.
2021-05-10 16:17:45 -0400 service CircuitBreakerTest has started 1 tasks: task f35ea31f9b574bc787fe5fad80f49a4d.
```
What's our expectation on the status change in Cloudformation?
I think after failed deployment, the Cloudformation status should be:
CREATE_IN_PROGRESS-->CREATE_FAILED-->ROLLBACK_IN_PROGRESS-->ROLLBACK_COMPLETE
And revert the cloudformation template to the latest configuration. By that way, user would know the ECS task update failed.
Contributor guide
Research direction
The issue names no repository files, tests, or entry points. Start by reviewing the reported ECS deployment and CloudFormation event sequence; done would be an agreed behavior for AWS::ECS::Service when the deployment circuit breaker fails, including rollback status handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100