kubeflow / kubeflow/trainer

Ship a default Grafana dashboard for Kubeflow Trainer

Open
#3,430 9 comments 0 reactions 1 assignee Claimed by @h0pers View on GitHub
area/observability kind/feature
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What you would like to be added?

Ship a default Grafana dashboard as part of the Helm chart that provides out-of-the-box visibility
into controller health and TrainJob lifecycle.

This revives [#1376](https://github.com/kubeflow/trainer/issues/1376), filed in 2021 for V1 but
closed as stale.

### Challenges

- Dashboard depends on which metrics are available (controller-runtime defaults today, custom
metrics once the companion metrics issue lands)
- Needs to cover two personas: **platform operators** (controller health, reconcile backlogs)
and **ML engineers** (TrainJob lifecycle, time-to-running, failure rates)
- Should be gated in Helm values to avoid installing for clusters without Grafana

### Why is this needed?

### Current State

- No dashboard shipped with Trainer
- Operators must manually discover available metrics, write PromQL, and build panels
- Peer projects (Kueue, Argo Workflows) bundle dashboards - Trainer is a gap

### Related Issues

- https://github.com/kubeflow/trainer/issues/1376
- https://github.com/kubeflow/trainer/issues/1340
- https://github.com/kubeflow/trainer/issues/2648
- https://github.com/kubeflow/trainer/issues/2779

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.