[ECS] [issue]: when ecs-agent abruptly disconnects running service tasks are never rescheduled to another CI
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
when ecs-agent abruptly disconnects (say due to underlying ec2 hardware failure) running service tasks are never rescheduled to another available CI
**Which service(s) is this request for?**
ECS - EC2 launch type
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
when ecs-agent abruptly disconnects (say due to underlying ec2 hardware failure) running service tasks are never rescheduled to another CI
To reproduce the issue I launched a couple of CIs in cluster and created a service with a handful of tasks. Then I did below on one of the CIs with some running service tasks
```
sudo docker stop ecs-agent && sudo service ecs stop
```
CI appears with attribute `agent connected = false` and tasks which were scheduled to the CI show desired state as `stopped` and last as `running`.
Ideally ecs control plane should detect that ecs-agent is down (may be by mechanism of heart beats as k8s does) and taint CI and reschedule running service tasks to another CI
**Are you currently working around this issue?**
reboot/terminate failed instance or restart ecs-agent if instance is still accessible
**Additional context**
n/a
**Attachments**
n/a
Contributor guide
Research direction
The issue names no repository files, tests, or implementation entry points. Reproduce it on an ECS EC2 cluster by stopping ecs-agent with `sudo docker stop ecs-agent && sudo service ecs stop`; done means service tasks from the disconnected CI are rescheduled onto another available CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100