aws / aws/containers-roadmap

Wrong desired count

Open
#525 0 comments 0 reactions 0 assignees View on GitHub
ECS
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Summary
Desired count is set wrong when an appautoscaling policy is triggered during deployment

### Description
Desired count is set to the sum of `ACTIVE + PRIMARY + SCALING ADJUSTMENT` when a policy is triggered while a deployment is running.

### Expected Behavior
Set correct desired count

### Environment Details
ecs instance ami amzn2-ami-ecs-hvm-2.0.20190614-x86_64-ebs (ami-04a084a6d17d9816e)

agent version 1.29.0

##### Deployment config
Minimum healthy percent: 100
Maximum percent: 200

##### AutoScaling config
Minimum tasks: 6
Maximum tasks: 24

SCALE-UP-service/prod/service-prod: CPUUtilization >= 99
Policy type: Step scaling
For alarm: service-prod_CPUUtilization
Take the action:
Add 1 tasks when 99 <= CPUUtilization

SCALE-DOWN-service/prod/service-prod: CPUUtilization >= 99
Policy type: Step scaling
For alarm: service-prod_CPUUtilization
Take the action:
Remove 1 tasks when <= CPUUtilization 99

### Supporting Log Snippets
(from ECS/My Service/Events)
autoscaling activity before deployment
```
2019-07-10 15:29:44 +0200 Message: Successfully set desired count to 10. Change successfully fulfilled by ecs. Cause: monitor alarm ECS_Service-service-prod_CPUUtilization in state ALARM triggered policy SCALE-UP-service/prod/service-prod
2019-07-10 15:29:48 +0200 service service-prod has started 1 tasks: task 269a6d27375c4d8b8def40061a341456.
2019-07-10 15:30:02 +0200 service service-prod has reached a steady state.
```
deployment begins here
```
2019-07-10 15:31:03 +0200 service service-prod has started 10 tasks
```
autoscaling policy is triggered
```
2019-07-10 15:31:03 +0200 Message: Successfully set desired count to 21. Change successfully fulfilled by ecs. Cause: monitor alarm ECS_Service-service-prod_CPUUtilization in state ALARM triggered policy SCALE-UP-service/prod/service-prod
2019-07-10 15:31:50 +0200 service service-prod has started 3 tasks: task f0cc7d528c294519984069c278fd0582 task 80e267e653f84f9e836540e17f394472 task e381f9755bc8406d8d8a3f9cce61e9b2.
2019-07-10 15:32:04 +0200 service service-prod was unable to place a task because no container instance met all of its requirements. The closest matching container-instance da22b00b-0b94-4c9f-a794-74859595e552 has insufficient CPU units available. For more information, see the Troubleshooting section.

... more of started/stopped tasks till deployment is completed
```

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are identified. Start by reviewing the reported ECS deployment and autoscaling event sequence, including the desired-count changes from 10 to 21, then determine the expected count during deployment; done means the service sets the correct desired count without adding the deployment adjustment twice.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.