kubeflow / kubeflow/trainer

Add GPU utilization Grafana dashboard with dcgm-exporter metrics

Open
#3,803 5 comments 0 reactions 1 assignee Claimed by @sanskar-singh-2403 View on GitHub
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What you would like to be added?

Add a Grafana dashboard for TrainJob GPU utilization using [dcgm-exporter](https://github.com/NVIDIA/dcgm-exporter) metrics. This would complement the controller health dashboard added in #3653 by providing visibility into GPU resource usage during training.

The dashboard could be modeled after the [NVIDIA dcgm-exporter dashboard](https://github.com/NVIDIA/dcgm-exporter/blob/main/grafana/dcgm-exporter-dashboard.json) and adapted for TrainJob workloads.

It should follow the same `grafanaDashboard` selection pattern introduced in #3653, allowing users to enable it individually or via `grafanaDashboard.defaultEnabled`.

### Why is this needed?

GPU utilization is a key metric for ML training workloads. Users running TrainJobs on GPU nodes need visibility into GPU memory usage, compute utilization, and temperature to optimize training performance and cost. Currently, the Grafana dashboard in #3653 only covers controller health and reconciliation metrics.

Ref: https://github.com/kubeflow/trainer/pull/3653#discussion_r3598455920

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.