aws / aws/containers-roadmap

[ECS] [Feature Request]: with longer stoptimeout, need "Force Stop" to kill rogue tasks/containers

Open
#2,320 0 comments 5 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
The new StopTimeout that allows us to set interval between SIGTERM and SIGKILL to be longer than 2 minutes is great.
We set it for 24 hours for our batch jobs and it works great.
However, we have run into issues where we know that that containers have received a SIGTERM and started draining (Desired State: Stopped), but the container has run into a busy wait loop (looking at their logs).
In these cases we want to "Force stop" the container/task.
It would be nice to have this features within the ECS gui under "ECS->->Tasks" as "Stop Selected Forced".
Right now we way we deal with this problem is with automation use SSM to kill the offending containers using the "docker kill" command.
Reference: https://aws.amazon.com/blogs/containers/improvements-to-amazon-ecs-task-launch-behavior-when-tasks-have-prolonged-shutdown/

**Which service(s) is this request for?**
ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Help us and other customers better manage rogue tasks which need to be force killed before "stopTimeout" has expired.

**Are you currently working around this issue?**
How are you currently solving this problem?
We built a helper program in python that finds tasks where desiredStatus=="STOPPED" and "lastStatus" == "RUNNING", find the underlying containers and ec2 instances and then use SSM to run "docker kill" on each server.

**Additional context**
If you want to discuss this issue, you can reach out to me at ashish.desai@zillasecurity.com

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the ECS console path ECS->->Tasks and the reported StopTimeout behavior. The issue also describes a Python helper that finds tasks with desiredStatus STOPPED and lastStatus RUNNING and uses SSM to run docker kill. Done means selected ECS tasks can be force-stopped before StopTimeout expires.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.