aws / aws/containers-roadmap

[ECS / Codepipeline] [request]: ECS Circuit Breaker should signal failures to CodePipeline

Open
#1,399 0 comments 0 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
We use CodePipeline to build our Docker images and then deploy them to ECS using a rolling service update. We recently enabled ECS's Circuit Breaker, and while the circuit breaker works as expected and terminates a deployment if it will never succeed, CodePipeline's deployment stage remains in the "In Progress" state even after the circuit breaker kicked in.

What I'm seeing:
1. CodePipeline enters the deploy stage and issues an UpdateService call to ECS with a new task definition
2. ECS launches 10 tasks, they all fail, and the circuit breaker terminates the deployment. This takes about ~20 minutes to start and stop all 10 tasks.
3. Codepipeline's deploy stage remains in the "In progress" state for another 40 minutes until it eventually times out at the 1 hour mark.
4. All further deployments are blocked until Codepipeline's deploy stage times out

Expected / desired behavior:
1. CodePipeline enters the deploy stage and issues an UpdateService call to ECS with a new task definition
2. ECS launches 10 tasks, they all fail, and the circuit breaker terminates the deployment. This takes about ~20 minutes to start and stop all 10 tasks.
3. Codepipeline's deploy stage is failed as soon as the circuit breaker terminates the deployment.

**Which service(s) is this request for?**
ECS and Codepipeline

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
If a developer pushes bad code and ECS tasks fail, the circuit breaker will terminate the deployment as desired. Codepipeline should be able to detect this and mark a deployment as failed. The developer shouldn't have to wait 60 minutes to be able to run a new deployment, especially when the circuit breaker terminates a deployment after ~20 minutes.

**Are you currently working around this issue?**
Not really. Just waiting around for CodePipeline to mark a deployment as failed.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the ECS rolling service update and CodePipeline deploy-stage behavior described in the issue, including how the ECS circuit breaker terminates failed deployments. Done means CodePipeline detects that termination and marks the deploy stage failed promptly instead of waiting for the one-hour timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.