[ECS] [feature]: Update ListTasks API to accept LastStatus argument
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Currently you can only list tasks by DesiredStatus of `RUNNING | PENDING| STOPPED`. Unfortunately, PENDING will never return anything as ECS never sets `DesiredStatus` to [PENDING](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ListTasks.html#ECS-ListTasks-request-desiredStatus), removing most of the value from the filter. However, if we can query by LastStatus we can monitor tasks that are PENDING.
**Which service(s) is this request for?**
ECS and Fargate
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We have found an issue where ECS Agent reports healthy but is in a bad state, most likely because of dockerd. As a result, the scheduler will put tasks on the a node that isn't healthy. We have found that the task stays in PENDING status indefinitely (we have witness over 2 days). This means ECS doesn't check on the tasks or reschedule them.
While #759 will help with this issue it doesn't do enough as we will still have to check every container instance. For us to monitor this with the current API we need to:
1. List all tasks with desired state RUNNING
2. Describe each task and check LastStatus
3. Describe the ContainerInstance and get InstanceID
4. SetInstance to Unhealthy (drain, etc)
We run a very large cluster with many tasks on it, describing all tasks doesn't scale nor does looking at every one of our many instances. Instead we would like to ListTasks by `LastStatus` then only have to map those individual tasks to a InstanceID .
**Are you currently working around this issue?**
We are not. When we think we have a bad instance we deploy a dummy service that goes onto every node and see where the longstanding PENDING tasks are. Its easier than describing every task/container instance.
Contributor guide
Research direction
Start by reviewing the AWS ECS ListTasks API documentation and its DesiredStatus and LastStatus parameters. Confirm the requested filtering behavior for ECS and Fargate, then define done as ListTasks accepting LastStatus and returning matching tasks without requiring per-task descriptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- api, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100