aws / aws/containers-roadmap

[ECS] [request]: Make ECS Scheduled Tasks Reliable

Open
#1,705 0 comments 18 reactions 1 assignee Claimed by @vibhav-ag View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
ECS Scheduled Tasks lose reliability under scale and do not provide easily actionable notices of failure. I would like ECS to:

1. Make RunTask API calls (ECS Scheduled Tasks are cloudwatch alarms that target ECS RunTask) reliable. They should not fail for RESOURCE:MEMORY or RESOURCE:CPU if the cluster is capable of provisioning additional compute.
2. When RunTask API calls fail, events in the "aws.ecs" namespace should be emitted to eventbridge, perhaps as "ECS Task State Change" messages with appropriate metadata set.

**Which service(s) is this request for?**
ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We have scheduled jobs setup as ECS Scheduled Tasks. One particular ECS cluster dedicated to this currently has 283 scheduled jobs that are invoked around 30k times per day. In the past 30 days the failure rate of these calls is 0.12%, or 2 9's of reliability.

The failed API calls are only visible by either:
1. searching CloudTrail
2. Setting up log alerting to find an absence of expected ECS task events.

The combination of reliability and observability is becoming a bigger problem as we scale up our use of ECS Scheduled Tasks.

The cluster this happens on has a capacity provider using an ASG that is allowed to scale out to 64 instances. We see these failures when the cluster only has a few instances. The RunTask API call seems to not respect the ECS orchestration and does not wait for capacity to provision the way what ECS services do.

I would like these issues with the ECS Scheduled Tasks product to be solved. Make RunTask reliable and also give me ECS Task events for this failure case so that i can create automation to retry.

**Are you currently working around this issue?**
I am currently getting a feed of failed tasks from semi-daily cloudtrail searches so that I can evaluate what may need to be re-run, hours after the fact. I also have an open case with AWS Enterprise Support and am posting this to help get the service team motivated on recommendation of the rep servicing my ticket.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.