Add GPU utilization Grafana dashboard with dcgm-exporter metrics
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
### What you would like to be added?
Add a Grafana dashboard for TrainJob GPU utilization using [dcgm-exporter](https://github.com/NVIDIA/dcgm-exporter) metrics. This would complement the controller health dashboard added in #3653 by providing visibility into GPU resource usage during training.
The dashboard could be modeled after the [NVIDIA dcgm-exporter dashboard](https://github.com/NVIDIA/dcgm-exporter/blob/main/grafana/dcgm-exporter-dashboard.json) and adapted for TrainJob workloads.
It should follow the same `grafanaDashboard` selection pattern introduced in #3653, allowing users to enable it individually or via `grafanaDashboard.defaultEnabled`.
### Why is this needed?
GPU utilization is a key metric for ML training workloads. Users running TrainJobs on GPU nodes need visibility into GPU memory usage, compute utilization, and temperature to optimize training performance and cost. Currently, the Grafana dashboard in #3653 only covers controller health and reconciliation metrics.
Ref: https://github.com/kubeflow/trainer/pull/3653#discussion_r3598455920
Contributor guide
Assessment
This issue has not been assessed yet.