aws / aws/containers-roadmap

[ECS] [request]: remove task-level memory requirement from Service Connect

Open
#2,838 0 comments 4 reactions 1 assignee Claimed by @radsn View on GitHub
ECS Proposed Service Connect
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

## Service Connect requires task-level memory, breaking bin-packing strategies that worked with App Mesh

### Tell us about your request: What do you want us to build?

Allow ECS Service Connect to work **without** requiring the task-level `memory` parameter, supporting container-level `memoryReservation` only (like App Mesh did). Alternatively, provide a deployment mechanism that doesn't require pre-allocating 2x task capacity for zero-downtime rolling updates.

### Which service(s) is this request for?

ECS on EC2 with Service Connect

### Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

**What we're trying to do:**
Migrate from App Mesh to Service Connect before App Mesh discontinuation (September 30, 2026) while maintaining efficient resource utilization on ECS EC2 with bin-packing strategies.

**Why it's hard:**
Service Connect **requires** setting task-level `memory`, which acts as a hard limit that triggers OOM kills even when the EC2 instance has available memory. This breaks bin-packing strategies that relied on App Mesh's container-level `memoryReservation` approach.

**App Mesh (worked efficiently):**
- Tasks used only container-level `memoryReservation`
- ECS scheduled based on reservation
- Containers could burst beyond reservation using available instance memory
- No task-level hard limit = no forced headroom during deployments

**Service Connect (forces waste):**
- **Requires** task-level `memory` parameter
- Task-level `memory` = hard limit, OOM kills containers even if instance has free RAM
- Rolling deployments with `maximumPercent=200` require pre-allocating space for 2x tasks
- Cannot utilize available instance memory efficiently

**Real production impact:**
- 50% of instances require upsizing to accommodate deployment headroom
- 60-88% wasted capacity on instances that technically "fit" but cannot utilize available memory
- Significant cost increases (50-100% per instance) for the same workload

**Example: Small instance bin-packing**
Instance memory: ~2 GB
Current workload: 2 tasks × ~800 MB = 1.6 GB actual usage (fits today)
With Service Connect deployment (200% max): 4 tasks × ~800 MB = 3.2 GB required
Result: Must upsize to next instance size (100% cost increase)

**Example: Larger instance with low utilization**
Instance memory: ~8 GB
Task-level memory required: ~1.5 GB per task
Running tasks: 2
Deployment headroom reserved: 2 additional tasks × 1.5 GB = 3 GB
Wasted capacity: ~5 GB (60%+ of paid memory sits unused)

**Why this matters:**
1. App Mesh discontinuation forces migration - we have no choice
2. Service Connect is the recommended migration path
3. The task-level memory requirement is architecturally incompatible with efficient bin-packing
4. Customers are forced to pay for significantly more capacity to run the same workload

**Impact of not solving this:**
- Forced instance upsizing (50-100% cost increase for affected instances)
- 60-88% wasted capacity on instances that technically "fit"
- Service Connect becomes materially worse than the App Mesh solution it's replacing
- Customers may explore non-AWS service mesh alternatives

### Are you currently working around this issue? How are you currently solving this problem?

We have no satisfactory workaround:

**Evaluated options (all unsatisfactory):**
- ❌ **Accept the cost increase**: Upsize instances and accept 60-88% wasted capacity
- ❌ **Reduce `maximumPercent` to 150%**: Reduces waste but increases deployment risk and still requires upsizing many instances
- ❌ **Reduce task density**: Makes waste even worse
- ❌ **VPC Lattice**: Significant complexity increase, higher cost, overkill for ECS-only use cases

**Current state:** Still on App Mesh, evaluating whether to accept the Service Connect cost penalty or explore alternative solutions.

### Additional context: Anything else we should know?

**Environment:**
- **ECS Launch Type:** EC2
- **Deployment Strategy:** Rolling updates with `minimumHealthyPercent=100`, `maximumPercent=200`
- **Previous Solution:** App Mesh with container-level `memoryReservation` only (no task-level memory)

**Why container-level memoryReservation worked:**
- ECS scheduler used `memoryReservation` for placement decisions
- Containers could use additional memory when available (bursting)
- No hard task-level cap meant deployments didn't require pre-allocated headroom
- Efficient bin-packing: 85-95% utilization vs 12-40% with Service Connect

**What would solve this:**
1. **(Preferred)** Make task-level `memory` optional for Service Connect, allow container-level `memoryReservation` only configuration by asking the user to input (through the task definition or the service configuration) the container-level `memoryReservation` for the Service Connect-injected envoy proxy container.
2. Smart deployment headroom calculation based on actual instance availability rather than requiring 2x pre-allocation
3. Document a migration path from App Mesh that preserves bin-packing efficiency
4. Don't retire App Mesh before having all of its features implemented by Service Connect first

This architectural constraint makes Service Connect economically worse than App Mesh for memory-constrained workloads, which undermines the migration path for customers forced off App Mesh.

### Attachments

**Measured impact pattern across production workloads:**
- Instances with 2-4 GB memory: 75-85% waste, typically require upsizing
- Instances with 4-8 GB memory: 30-60% waste, some require upsizing
- Instances with 8+ GB memory: 60-88% waste even when tasks fit
- Common pattern: deployment headroom (2x tasks) cannot be accommodated without significant over-provisioning

### Notes

The percentage values are calculated using our actual setup, they may vary of course but the math is more or less the same for similar use-cases.

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.