bottlerocket-os / bottlerocket-os/bottlerocket

Add `warm-pools-check` to `settings.ecs` for ECS warm pool lifecycle support

Open
#4,888 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

## Description

The ECS agent supports `ECS_WARM_POOLS_CHECK=true` (since agent v1.59.0) to enable proper warm pool lifecycle management with managed termination protection. This setting is required for ECS to release scale-in protection when tasks complete, allowing instances to return to the ASG warm pool (Stopped state).

Currently, Bottlerocket ECS variants do not expose this setting in `settings.ecs`. Attempting to set it in user data causes a boot failure:

```
[FAILED] Failed to start Bottlerocket userdata configuration system.
Json deserialize error: unknown field `warm-pools-check`, expected one of `cluster`,
`instance-attributes`, `allow-privileged-containers`, `logging-drivers`, `loglevel`,
`enable-spot-instance-draining`, `image-pull-behavior`, `container-stop-timeout`,
`task-cleanup-wait`, `metadata-service-rps`, `metadata-service-burst`, `reserved-memory`,
`image-cleanup-wait`, `image-cleanup-delete-per-cycle`, `image-cleanup-enabled`,
`image-cleanup-age`, `backend-host`, `awsvpc-block-imds`, `enable-container-metadata`
```

## Impact

Without this setting, ECS `enableManagedTerminationProtection` never releases scale-in protection after tasks complete. Instances pulled from the warm pool for burst workloads remain Running indefinitely at full compute cost, even with zero tasks. For GPU instances (g4dn.xlarge at $0.587/hr), this results in significant unnecessary spend.

## Proposed change

Add `warm_pools_check: bool` to `ECSSettingsV1` in `bottlerocket-settings-sdk/bottlerocket-settings-models/settings-extensions/ecs/src/lib.rs`:

```rust
#[model(impl_default = true)]
pub struct ECSSettingsV1 {
// ... existing fields ...
enable_container_metadata: bool,
warm_pools_check: bool, // NEW
}
```

And map it to `ECS_WARM_POOLS_CHECK=true` in the ECS agent systemd environment configuration.

Default should be `false` (consistent with Amazon Linux behavior where it is opt-in).

A settings migration will be needed for upgrades (similar to `v1.12.0/add-k8s-autoscaling-warm-pool-setting`).

## Reference

- ECS warm pool docs: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-warm-pool.html
- ECS agent variable: `ECS_WARM_POOLS_CHECK` (agent >= 1.59.0)
- Kubernetes equivalent: `settings.autoscaling.should-wait` (added in Bottlerocket v1.12.0 via a different mechanism)
- Affected variants: `aws-ecs-1`, `aws-ecs-2`, `aws-ecs-1-nvidia`, `aws-ecs-2-nvidia`
- Tested on Bottlerocket 1.62.x (current) - setting not present

## Environment

- Bottlerocket variant: `aws-ecs-2-nvidia`
- Instance type: g4dn.xlarge
- ECS Capacity Provider with `enableManagedTerminationProtection: true`
- ASG warm pool with `poolState: Stopped`, `reuseOnScaleIn: true`

Contributor guide

Open the contributing guide

Research direction

Start in bottlerocket-settings-sdk/bottlerocket-settings-models/settings-extensions/ecs/src/lib.rs and inspect how ECS settings are mapped into the ECS agent systemd environment. Review the v1.12.0/add-k8s-autoscaling-warm-pool-setting migration as a guide for the required upgrade migration. Done means warm_pools_check defaults to false, accepts the setting for the listed ECS variants, and maps it to ECS_WARM_POOLS_CHECK.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud, operating-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.