aws / aws/containers-roadmap

[ECS] [request]: ECS Deployment gets stuck if target groups deleted

Open
#1,884 0 comments 2 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
If an ECS service is attached to a load balancer via target groups, and the target groups are deleted (e.g. to change properties about them, such as port or LB attachment) and then a deployment is triggered to the ECS service, it gets stuck and never completes.

**Which service(s) is this request for?**
ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We manage our infrastructure using Terraform and while re-configuring ports and target groups for an ECS service, some of the existing target groups the ECS service was attached to were deleted. As part of the same terraform change, we also triggered an update to the ECS service via an updated task definition. ECS started a new task with the new task definition, and it got registered with the new target groups and reported healthy. However, the old task was never reaped and the service was never marked as attaining steady state.

We opened an AWS support ticket for this and were advised to manually kill the old task. We did that and only at that point did ECS recognize that the old target groups were gone. ECS then logged that it was trying to drain targets but by that point, the old task was already gone. It wasn't until we manually ran a "force new deployment" on the service with no other changes that the service was marked as entering steady state.

I understand that this sort of scenario doesn't likely happen often; however, there's definitely room for improvement in the platform to better handle this scenario.

**Are you currently working around this issue?**
We're working around this issue by manually killing the old/inactive task and running an update with no changes aside from "force new deploy."

**Additional context**
N/A

**Attachments**
N/A

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or implementation entry points are identified; this is an AWS ECS platform behavior report. Start by reproducing the deployment with deleted target groups and an updated task definition, then compare task draining and steady-state behavior. Done means the service detects the missing target groups and completes deployment without manually killing tasks or forcing another deployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.