aws / aws/containers-roadmap

[ECS] [request]: Prioritize Daemon task scheduling above Replica tasks

Open
#428 25 comments 110 reactions 1 assignee Assigned to @AbhishekNautiyal View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

**Tell us about your request**
When our Replica ECS Services scale up and we launch new ECS Container Instances (from an ASG, scaling on *Reservation metrics), sometimes the Replica tasks are launched on the instance fast enough that our Daemon Services do not have a chance to launch on these new hosts. If these replicas use enough CPU/memory, there may not be room for the Daemon services to run.

For example, we have a few daemon services to collect host-level metrics and forward log files. We want these to run on every host. Periodically we see that hosts have been saturated with Replica tasks and there's no room for the Daemon tasks. This means we lack monitoring and visibility into these hosts.

**Which service(s) is this request for?**
This could be ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

Every host should have Daemon tasks provisioned on it, reliably.

**Are you currently working around this issue?**
We manually go into the ECS console, review all the (possibly hundreds) of hosts to identify which one is missing Daemon services, then run "Stop Task" for one of the replica tasks on that host in order for Daemons tasks to have room to launch.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.