[ECS Managed Instances] [request]: Enable detailed instance monitoring and expose ENA network metrics for Managed Instances
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
For ECS Managed Instances, provide both of the following:
1. A way to enable EC2 detailed (1-minute) monitoring on the instances a capacity provider launches.
2. ENA network performance metrics published as CloudWatch metrics for those instances: `bw_in_allowance_exceeded`, `bw_out_allowance_exceeded`, `pps_allowance_exceeded`, `conntrack_allowance_exceeded`, `linklocal_allowance_exceeded`.
**Which service(s) is this request for?**
ECS (Amazon ECS Managed Instances)
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We are diagnosing intermittent multi-second stalls on a network-heavy service running on Managed Instances, and we cannot determine whether the instances are hitting ENA allowance limits. Two things block us.
**No detailed monitoring.** `NetworkIn` and `NetworkOut` are available only at 5-minute basic-monitoring granularity, and the capacity provider offers no way to enable detailed monitoring. Our workload is bursty at sub-second scale, so a 5-minute average is not merely imprecise, it is misleading: it reports a few MB/s while the instance is briefly at line rate. Selecting the `Maximum` statistic does not help, because each datapoint is already an average over its period.
**ENA counters are unreachable from inside the task.** The usual approach is the CloudWatch agent's `ethtool` plugin reading ENA driver counters. On Managed Instances the task ENI is a trunk-ENI VLAN sub-interface with no ENA device visible in the task network namespace, which we confirmed via ECS Exec. So the counters cannot be collected from a sidecar, and Managed Instances provides neither host access nor daemon placement to collect them from the host.
The result is that a workload on Managed Instances has strictly less network observability than the same workload on self-managed EC2, with no way to close the gap. We verified these counters are present and valuable on our self-managed EC2 fleets, where they showed inbound bandwidth allowance being exceeded during 45% of five-minute intervals on hosts that appeared roughly 80% idle at minute granularity. That is exactly the signal we are missing on Managed Instances.
**Are you currently working around this issue?**
Only partially, and no workaround is sufficient:
* Sampling `/proc/net/dev` and `/proc/net/snmp` from inside the task via ECS Exec. This yields task-level byte counters and TCP retransmits but not ENA allowance counters, and it is manual rather than continuous.
* Inferring saturation from downstream service latency, which is the guesswork we are trying to eliminate.
* Running the affected fleets on self-managed EC2 instead, which sacrifices managed patching.
**Additional context**
There is precedent for adding per-instance metrics to Managed Instances: #2734 (GPU metrics support for ECS Managed Instances) was accepted and completed. The equivalent detailed-monitoring ask for EKS is #2660. #2788 (instance-level metadata and daemon service support) would be one mechanism that could satisfy part of this request.
**Attachments**
None.
Contributor guide
Research direction
No repository files or tests are identified. Start by reviewing the related roadmap issues #2734, #2660, and #2788, along with the CloudWatch agent ethtool limitation described here. Done means ECS Managed Instances support both EC2 detailed monitoring and the listed ENA allowance metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100