[ECS] [request]: functional CPU throttling stats in the task metadata endpoint
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
[The documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html) says that `${ECS_CONTAINER_METADATA_URI_V4}/task/stats` returns docker [ContainerStats](https://docs.docker.com/reference/api/engine/version/v1.30/#tag/Container/operation/ContainerStats), which includes CPU throttling metrics at `cpu_stats.throttling_data`. While this data is present, I have [experimentally verified](https://github.com/prometheus-community/ecs_exporter/issues/41#issuecomment-2401119026) that it is never incremented from 0 even when throttling is occurring.
Quoting that comment here:
> I ran tasks with both ecs_exporter and an alpine sidecar running ["/bin/sh", "-c", "yes > /dev/null"] (i.e. chewing up a lot of CPU) on Fargate and EC2. They both had less than 1 vCPU allocated, Fargate at the task level and EC2 at the container level. The CPU-seconds metrics for both were definitely increasing slower than real time passed, indicating that throttling was occurring. The builtin CloudWatch graphs available in the AWS console also indicated that these services were using all available CPU.
>
> But the throttling container stats remained at 0. I'm not sure why, but regardless I think this is a dead end without action from AWS.
@singholt says that this is due to [implementation problems in the ECS agent](https://github.com/aws/amazon-ecs-agent/issues/4473#issuecomment-3881298834). ECS agent needs work to report this data properly.
**Which service(s) is this request for?**
ECS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
This data would be nice to have to help better make service operators aware of e.g. when their Fargate task CPU size is too low. Any kind of sampling of current CPU utilization (like you can do with CloudWatch metrics, or with the functional parts of ContainerStats) can miss throttling events (because you sampled only when throttling was not occurring), whereas these metrics would provide a definitive record of throttling having occurred.
**Are you currently working around this issue?**
No known workaround.
Contributor guide
Assessment
This issue has not been assessed yet.