aws / aws/eks-charts

GPU metrics not collected by aws-cloudwatch-metrics

Open
#1,097 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Mustache
Stars
1.3k
Forks
1.1k
Avg merge
22m
Merged PRs (30d)
5

Description

**Describe the bug**

I've setup the aws-cloudwatch-metrics through the helm chart linked here, I've also set the `image.tag=1.300037.0b583`, because it seems that the GPU metrics should be collected by default starting from `1.300034.0` according to this [link](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-EKS.html).

Also the RBAC permissions have been manually updated to include services: https://github.com/aws/eks-charts/pull/1095 as well as I've explicitly set `enhancedContainerInsights.enabled=true` (and fixed the documentation for this value [here](https://github.com/aws/eks-charts/pull/1094)).

I still can't see the metrics in ContainerInsights and I start to believe, that I have to add additional settings to the ConfigMap to explicitly enable the GPU metrics collection. Can someone confirm this, or should GPU metrics collection would out of the box?

**Steps to reproduce**

Install aws-cloudwatch-metrics on a EKS cluster with GPU nodes (e.g. g5.xlarge). Check CloudWatch for GPU metrics.

**Expected outcome**

I'd expect the GPU metrics to show up in CloudWatch

**Environment**

* Chart name: aws-cloudwatch-metrics
* Chart version: 0.0.11
* Kubernetes version: 1.29.3-eks-adc7111
* Using EKS (yes/no), if so version? 1.29.3-eks-adc7111

**Additional Context**:

I've successfully set up the metrics collection for GPU metrics on EC2 instances before, but it doesn't seem to work on EKS using this chart.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the aws-cloudwatch-metrics Helm chart on an EKS 1.29.3 cluster containing a g5.xlarge GPU node, using chart version 0.0.11 and enhancedContainerInsights.enabled=true. Inspect the chart's ConfigMap and RBAC settings to determine whether GPU collection needs explicit configuration. Done means the required configuration is identified and GPU metrics appear in CloudWatch, or the limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes
Domain
cloud, devops, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.