aws / aws/containers-roadmap

[ECS] [request]: Allow configuration on when ECS kills task after made unhealthy by ALB

Open
#1,271 6 comments 77 reactions 1 assignee Claimed by @vibhav-ag View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Our applications can have unpredictable spike in traffic. During this time the backends get overwhelmed with request and start rejecting new connections. This results in failing health checks and ECS killing the task, countering any scaling ECS is performing. Instead we are looking for it to remain active, finish its requests, and give it a chance to become healthy again while the service is scaling out to handle the load. (essentially a circuit breaker in ECS).

ALB doesn't automatically kill the targets as they leave it to the service to decide (ECS or ASG for example)

**Which service(s) is this request for?**
Fargate, ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Essentially have a circuit breaker in ECS. Allow it to fail health checks but also give the tasks a chance to recover. During large scale events the failing health checks make scaling even harder to perform as many old tasks could help with the load are recycled.

**Are you currently working around this issue?**
Teams have tested this in EC2 and are thinking of moving there. The way they establish it is by:
1. Setting ASG health check to instance health check
2. Allows ALB to remove target when overwhelmed and failing ALB Health Check then have a local check checking some other form of healthiness (maybe another port that has metrics etc) and when it fails mark the instance as unhealthy.

**Additional context**

**Attachments**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.