Add additional panels to Grafana dashboard
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
### What you would like to be added?
Add additional panels to the Grafana dashboard introduced in #3653. The following metrics are already exposed by the controller and can be added without code changes:
- **CPU usage** - `process_cpu_seconds_total` (from the Go process collector)
- **Controller uptime** - `process_start_time_seconds` (from the Go process collector)
- **K8s API client call rates** - `rest_client_requests_total` (from client-go)
- **K8s API client latency** - `rest_client_request_duration_seconds` (from client-go)
- **K8s API client errors** - `rest_client_requests_total{code=~"4..|5.."}` (from client-go)
- **K8s API client throttling** - `rest_client_rate_limiter_duration_seconds` (from client-go)
The following would require additional work outside the dashboard:
- **Operator version/git SHA** - requires adding ldflags to the Dockerfile and registering a custom info metric
- **TrainJob/Runtime counts** - requires custom gauges in the controller or kube-state-metrics CustomResourceState configuration
### Why is this needed?
These metrics provide deeper visibility into the controller health and Kubernetes API interactions, helping operators debug performance issues and resource usage.
Ref: https://github.com/kubeflow/trainer/pull/3653#issuecomment-5080442109
Contributor guide
Assessment
This issue has not been assessed yet.