kubeflow / kubeflow/trainer

Add additional panels to Grafana dashboard

Open
#3,804 3 comments 0 reactions 1 assignee Claimed by @sanskar-singh-2403 View on GitHub
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What you would like to be added?

Add additional panels to the Grafana dashboard introduced in #3653. The following metrics are already exposed by the controller and can be added without code changes:

- **CPU usage** - `process_cpu_seconds_total` (from the Go process collector)
- **Controller uptime** - `process_start_time_seconds` (from the Go process collector)
- **K8s API client call rates** - `rest_client_requests_total` (from client-go)
- **K8s API client latency** - `rest_client_request_duration_seconds` (from client-go)
- **K8s API client errors** - `rest_client_requests_total{code=~"4..|5.."}` (from client-go)
- **K8s API client throttling** - `rest_client_rate_limiter_duration_seconds` (from client-go)

The following would require additional work outside the dashboard:

- **Operator version/git SHA** - requires adding ldflags to the Dockerfile and registering a custom info metric
- **TrainJob/Runtime counts** - requires custom gauges in the controller or kube-state-metrics CustomResourceState configuration

### Why is this needed?

These metrics provide deeper visibility into the controller health and Kubernetes API interactions, helping operators debug performance issues and resource usage.

Ref: https://github.com/kubeflow/trainer/pull/3653#issuecomment-5080442109

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.