[ECS] [bug]: Managed termination protection does not reliably remove scale-in protection from instances with only daemon tasks
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
### Tell us about your request
ECS managed termination protection (`managedTerminationProtection = ENABLED`) does not reliably remove scale-in protection from EC2 instances after all non-daemon tasks have drained.
Per the [documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/managed-termination-protection.html):
> "When all non-daemon tasks are stopped on an instance, Amazon ECS initiates the scale-in process and turns off scale-in protection for the EC2 instance."
> "Tasks that are run by a service that uses the DAEMON scheduling strategy are ignored and an instance can be terminated by cluster auto scaling even when the instance is running these tasks."
In practice, ECS sets `protectedFromScaleIn=True` when worker tasks are placed but does not call `SetInstanceProtection(False)` after workers drain, leaving daemon-only instances protected indefinitely.
### CloudTrail evidence
Instance `i-072d4027a290f93c5` on cluster `ai-compute-production`:
| Time | Caller | Action | Error |
|---|---|---|---|
| 2026-05-12 17:57 | ecs.amazonaws.com | `protectedFromScaleIn=True` | OK |
| 2026-05-12 18:13 | ecs.amazonaws.com | `protectedFromScaleIn=False` | OK |
| 2026-05-12 19:42 | ecs.amazonaws.com | `protectedFromScaleIn=True` | OK |
| **44 hours — no calls** | | | |
| 2026-05-14 15:52 | manual (CLI) | `protectedFromScaleIn=False` | OK |
| 2026-05-14 15:53-15:58 | ecs.amazonaws.com | `protectedFromScaleIn=False` (every minute) | OK |
After the worker task drained at ~19:42 on May 12, only the Datadog agent daemon remained. ECS did not call `SetInstanceProtection(False)` for 44 hours. Zero `RateExceeded` errors in CloudTrail — this is not API throttling.
After manual intervention removed protection, ECS resumed calling `SetInstanceProtection(False)` every minute, suggesting the manual action re-triggered ECS's evaluation loop.
A second instance (`i-03a617edaa7238a30`) showed the same pattern — ECS set `protectedFromScaleIn=True` and never called `False` until manual intervention.
### Configuration
- ECS capacity provider: `managedTerminationProtection = ENABLED`, `managedScaling = ENABLED`, `targetCapacity = 95`, `instanceWarmupPeriod = 120`
- Auto Scaling group: `protect_from_scale_in = true` (per documentation requirement)
- Daemon service: Datadog agent (`DAEMON` scheduling strategy)
- Worker services: Temporal activity workers (`REPLICA` scheduling strategy) with dynamic scaling via Lambda autoscaler
- ECS agent version: 1.103.0+
### Impact
ASG scaling activities show `Could not scale to desired capacity because all remaining instances are protected from scale-in`. Daemon-only instances accumulate, each costing ~$0.57/hr (`m6in.2xlarge`). Manual intervention is required to unblock scale-in.
### Workaround
Manually remove protection from daemon-only instances:
```bash
aws autoscaling set-instance-protection \
--auto-scaling-group-name \
--instance-ids \
--no-protected-from-scale-in
```
Or implement a scheduled Lambda (every 15 minutes) to find instances with only daemon tasks and `ProtectedFromScaleIn=True`, and remove protection automatically.
Contributor guide
Assessment
This issue has not been assessed yet.