aws / aws/containers-roadmap

[ECS] [issue]: when ecs-agent abruptly disconnects running service tasks are never rescheduled to another CI

Open
#2,040 0 comments 2 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
when ecs-agent abruptly disconnects (say due to underlying ec2 hardware failure) running service tasks are never rescheduled to another available CI

**Which service(s) is this request for?**
ECS - EC2 launch type

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
when ecs-agent abruptly disconnects (say due to underlying ec2 hardware failure) running service tasks are never rescheduled to another CI

To reproduce the issue I launched a couple of CIs in cluster and created a service with a handful of tasks. Then I did below on one of the CIs with some running service tasks

```
sudo docker stop ecs-agent && sudo service ecs stop
```

CI appears with attribute `agent connected = false` and tasks which were scheduled to the CI show desired state as `stopped` and last as `running`.

Ideally ecs control plane should detect that ecs-agent is down (may be by mechanism of heart beats as k8s does) and taint CI and reschedule running service tasks to another CI

**Are you currently working around this issue?**
reboot/terminate failed instance or restart ecs-agent if instance is still accessible

**Additional context**
n/a

**Attachments**
n/a

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files, tests, or implementation entry points. Reproduce it on an ECS EC2 cluster by stopping ecs-agent with `sudo docker stop ecs-agent && sudo service ecs stop`; done means service tasks from the disconnected CI are rescheduled onto another available CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.