aws / aws/containers-roadmap

[Fargate] [request]: Key-driven task scheduling and hot-swap replacement for stateful Fargate workloads

Open
#2,867 0 comments 9 reactions 0 assignees View on GitHub
ECS Fargate Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

**1. Key-driven task mode** — Today, Fargate services only support a flat `desiredCount`. We need a mode where a service accepts a dynamic map of `{ taskKey: workerCount }` and ensures the correct number of tasks per key. The key should be available to the task at runtime (e.g., via an environment variable). This would eliminate the need to manage a separate ECS service per logical partition.

**2. Hot-swap task replacement** — Fargate currently launches a replacement task and waits for it to be running before retiring the old one. However, "running" does not mean the new task has fully bootstrapped its in-memory state. We need a hot-swap mechanism where the new task boots alongside the old one, fully loads its state, and explicitly signals readiness — only then should load shift to it and the old task be drained. This would provide zero downtime with no in-memory state loss.

**Which service(s) is this request for?**

Fargate

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

We run latency-sensitive, high-cadence workloads on Fargate where each logical entity (e.g., a facility, a site, or a tenant) requires a dedicated task. Two examples:

#### Real-time State Tracking

- Continuously computes a state snapshot for a physical site every **5 seconds**. Each site has a large number of tracked assets, requiring a full Fargate task per site.
- **Zero tolerance for cold starts** — if state computation is interrupted for more than **20 seconds**, downstream systems receive stale data and automated decisions degrade.

#### Periodic Job Orchestration

- Manages job lifecycles per site on a **20-second** cycle — scheduling, monitoring, and completing short-lived units of work.
- Delays beyond **60 seconds** cause job backlogs and visible service degradation.

Today, there is no way to express "run one task per site" as a single service, and no way to ensure the replacement task is fully bootstrapped before the old one is retired — both of which are critical given our tight SLAs.

**Are you currently working around this issue?**

Yes. We built a custom orchestration layer that stitches together multiple AWS services to achieve this:

- **EventBridge + Lambda** — An EventBridge rule triggers a Lambda function every **1 minute**. The Lambda reads a dynamic key set, spins up new Fargate tasks for any newly added keys, and gracefully retires tasks whose keys no longer exist.
- **DynamoDB** — Stores the active key-to-task mapping and coordinates hot-swap state, ensuring the new task is fully bootstrapped before the old one is drained.
- **Fargate** — Runs the actual per-site workloads.

It works, but we are essentially building a custom key-aware scheduler and hot-swap coordinator on top of Fargate — significant undifferentiated overhead that could be a native capability.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the requested Fargate behavior and the stated EventBridge, Lambda, and DynamoDB workaround. Compare the key-to-task mapping and readiness/draining requirements with current AWS container-service capabilities. Done would require an agreed native design that meets the described per-key scheduling and zero-downtime state handoff goals.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.