GPU metrics not collected by aws-cloudwatch-metrics
- Dominant language
- Mustache
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 22m
- Merged PRs (30d)
- 5
Description
**Describe the bug**
I've setup the aws-cloudwatch-metrics through the helm chart linked here, I've also set the `image.tag=1.300037.0b583`, because it seems that the GPU metrics should be collected by default starting from `1.300034.0` according to this [link](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-EKS.html).
Also the RBAC permissions have been manually updated to include services: https://github.com/aws/eks-charts/pull/1095 as well as I've explicitly set `enhancedContainerInsights.enabled=true` (and fixed the documentation for this value [here](https://github.com/aws/eks-charts/pull/1094)).
I still can't see the metrics in ContainerInsights and I start to believe, that I have to add additional settings to the ConfigMap to explicitly enable the GPU metrics collection. Can someone confirm this, or should GPU metrics collection would out of the box?
**Steps to reproduce**
Install aws-cloudwatch-metrics on a EKS cluster with GPU nodes (e.g. g5.xlarge). Check CloudWatch for GPU metrics.
**Expected outcome**
I'd expect the GPU metrics to show up in CloudWatch
**Environment**
* Chart name: aws-cloudwatch-metrics
* Chart version: 0.0.11
* Kubernetes version: 1.29.3-eks-adc7111
* Using EKS (yes/no), if so version? 1.29.3-eks-adc7111
**Additional Context**:
I've successfully set up the metrics collection for GPU metrics on EC2 instances before, but it doesn't seem to work on EKS using this chart.
Contributor guide
Research direction
Start by reproducing the issue with the aws-cloudwatch-metrics Helm chart on an EKS 1.29.3 cluster containing a g5.xlarge GPU node, using chart version 0.0.11 and enhancedContainerInsights.enabled=true. Inspect the chart's ConfigMap and RBAC settings to determine whether GPU collection needs explicit configuration. Done means the required configuration is identified and GPU metrics appear in CloudWatch, or the limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, helm, kubernetes
- Domain
- cloud, devops, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100