aws / aws/containers-roadmap

[ECS] [request]: CAS, scaling in properly.

Open
#1,150 1 comment 5 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
ECS Cluster Auto Scaling (a.k.a CAS) does not scaling-in as I would like to.

Once I configured Capacity Provider with single AZ, the ASG started to scale out properly -- it responds to rapid desired service count changes fine, see Activity log from ASG below [1]. You will see, that it changed capacity from 14 to 26 in one shot. So far so good.

But when I update Desired service count to the lower value, Capacity provider updates ASG count by one instance, and for example changing from 26 to 2 instances takes ages. See ASG Activity history below [2]. It subtracts instances from ASG one by one, on every alarm trigger (so giving us around 1 instance less per minute), on scale in event, in order to meet configured Capacity provider reservation (100 in our case). I've even configured task definition and launch template to map cpu and mem reservation 1:1 (so single ECS task running on defined instance type will get all cpu and mem available on that instance type, having ECS agent reservevation in mind) to help ECS calculate it properly, but it changed nothing.

**Which service(s) is this request for?**
ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
-- nothing more to add --

**Are you currently working around this issue?**
The only workaround I found it to code ECS cluster auto scaling feature manually, with proper calculation of required EC2 instance count to handle the ECS workload, using whatever stack you like (Lambda? ECS service running in ops vpc? Its up to you)

**Additional context**
[1] ASG Activity history for **scale out** event:
```
At 2020-11-23T14:03:02Z a monitor alarm TargetTracking-mike-test-02-AlarmHigh-1938fef8-e8d6-4c9f-bcc4-4892b6193096 in state ALARM triggered policy AutoScaling-ECSManagedAutoScalingPlan-8ce3433b-5551-4568-ba4f-9736a074c229-1c9c9a5f-8961-44d1-8038-7790a49804ab changing the desired capacity from 14 to 26. At 2020-11-23T14:03:11Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 14 to 26.

or

At 2020-11-23T13:03:02Z a monitor alarm TargetTracking-mike-test-02-AlarmHigh-1938fef8-e8d6-4c9f-bcc4-4892b6193096 in state ALARM triggered policy AutoScaling-ECSManagedAutoScalingPlan-8ce3433b-5551-4568-ba4f-9736a074c229-1c9c9a5f-8961-44d1-8038-7790a49804ab changing the desired capacity from 2 to 14. At 2020-11-23T13:03:10Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 2 to 14.
```

[2] ASG Activity history for **scale in** event:
```
At 2020-11-23T12:27:37Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 26 to 25. At 2020-11-23T12:27:37Z instance i-01db2640b4123456 was selected for termination.

At 2020-11-23T12:28:20Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 25 to 24. At 2020-11-23T12:28:20Z instance i-0357e56c91123456 was selected for termination.

.
.
.

At 2020-11-23T12:47:39Z a monitor alarm TargetTracking-mike-test-02-AlarmLow-5b7b26fa-1883-4a57-8ea7-0681c3194939 in state ALARM triggered policy AutoScaling-ECSManagedAutoScalingPlan-8ce3433b-5551-4568-ba4f-9736a074c229-1c9c9a5f-8961-44d1-8038-7790a49804ab changing the desired capacity from 5 to 4. At 2020-11-23T12:47:48Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 5 to 4. At 2020-11-23T12:47:48Z instance i-056851a19468123456 was selected for termination.

At 2020-11-23T12:48:39Z a monitor alarm TargetTracking-mike-test-02-AlarmLow-5b7b26fa-1883-4a57-8ea7-0681c3194939 in state ALARM triggered policy AutoScaling-ECSManagedAutoScalingPlan-8ce3433b-5551-4568-ba4f-9736a074c229-1c9c9a5f-8961-44d1-8038-7790a49804ab changing the desired capacity from 4 to 3. At 2020-11-23T12:48:51Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 4 to 3. At 2020-11-23T12:48:51Z instance i-0c992011e37123456 was selected for termination.

At 2020-11-23T12:49:39Z a monitor alarm TargetTracking-mike-test-02-AlarmLow-5b7b26fa-1883-4a57-8ea7-0681c3194939 in state ALARM triggered policy AutoScaling-ECSManagedAutoScalingPlan-8ce3433b-5551-4568-ba4f-9736a074c229-1c9c9a5f-8961-44d1-8038-7790a49804ab changing the desired capacity from 3 to 2. At 2020-11-23T12:49:55Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 3 to 2. At 2020-11-23T12:49:55Z instance i-0776339e2ab123456 was selected for termination.
```

**Attachments**
-- none --

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files, tests, or entry points. Start by reviewing ECS Cluster Auto Scaling, Capacity Providers, and the supplied Auto Scaling Group activity history. Done would require a defined and validated resolution for scaling in from the requested capacity rather than removing one instance per alarm.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.