bottlerocket-os / bottlerocket-os/bottlerocket
Add `warm-pools-check` to `settings.ecs` for ECS warm pool lifecycle support
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
## Description
The ECS agent supports `ECS_WARM_POOLS_CHECK=true` (since agent v1.59.0) to enable proper warm pool lifecycle management with managed termination protection. This setting is required for ECS to release scale-in protection when tasks complete, allowing instances to return to the ASG warm pool (Stopped state).
Currently, Bottlerocket ECS variants do not expose this setting in `settings.ecs`. Attempting to set it in user data causes a boot failure:
```
[FAILED] Failed to start Bottlerocket userdata configuration system.
Json deserialize error: unknown field `warm-pools-check`, expected one of `cluster`,
`instance-attributes`, `allow-privileged-containers`, `logging-drivers`, `loglevel`,
`enable-spot-instance-draining`, `image-pull-behavior`, `container-stop-timeout`,
`task-cleanup-wait`, `metadata-service-rps`, `metadata-service-burst`, `reserved-memory`,
`image-cleanup-wait`, `image-cleanup-delete-per-cycle`, `image-cleanup-enabled`,
`image-cleanup-age`, `backend-host`, `awsvpc-block-imds`, `enable-container-metadata`
```
## Impact
Without this setting, ECS `enableManagedTerminationProtection` never releases scale-in protection after tasks complete. Instances pulled from the warm pool for burst workloads remain Running indefinitely at full compute cost, even with zero tasks. For GPU instances (g4dn.xlarge at $0.587/hr), this results in significant unnecessary spend.
## Proposed change
Add `warm_pools_check: bool` to `ECSSettingsV1` in `bottlerocket-settings-sdk/bottlerocket-settings-models/settings-extensions/ecs/src/lib.rs`:
```rust
#[model(impl_default = true)]
pub struct ECSSettingsV1 {
// ... existing fields ...
enable_container_metadata: bool,
warm_pools_check: bool, // NEW
}
```
And map it to `ECS_WARM_POOLS_CHECK=true` in the ECS agent systemd environment configuration.
Default should be `false` (consistent with Amazon Linux behavior where it is opt-in).
A settings migration will be needed for upgrades (similar to `v1.12.0/add-k8s-autoscaling-warm-pool-setting`).
## Reference
- ECS warm pool docs: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-warm-pool.html
- ECS agent variable: `ECS_WARM_POOLS_CHECK` (agent >= 1.59.0)
- Kubernetes equivalent: `settings.autoscaling.should-wait` (added in Bottlerocket v1.12.0 via a different mechanism)
- Affected variants: `aws-ecs-1`, `aws-ecs-2`, `aws-ecs-1-nvidia`, `aws-ecs-2-nvidia`
- Tested on Bottlerocket 1.62.x (current) - setting not present
## Environment
- Bottlerocket variant: `aws-ecs-2-nvidia`
- Instance type: g4dn.xlarge
- ECS Capacity Provider with `enableManagedTerminationProtection: true`
- ASG warm pool with `poolState: Stopped`, `reuseOnScaleIn: true`
Contributor guide
Research direction
Start in bottlerocket-settings-sdk/bottlerocket-settings-models/settings-extensions/ecs/src/lib.rs and inspect how ECS settings are mapped into the ECS agent systemd environment. Review the v1.12.0/add-k8s-autoscaling-warm-pool-setting migration as a guide for the required upgrade migration. Done means warm_pools_check defaults to false, accepts the setting for the listed ECS variants, and maps it to ECS_WARM_POOLS_CHECK.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, operating-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100