Add controller-level Prometheus metrics and ServiceMonitor

### What you would like to be added?

Add custom Prometheus metrics to the Kubeflow Trainer controller manager and ship a `ServiceMonitor`
CR in the Helm chart for Prometheus Operator scraping.

### Challenges

- Defining a metric naming convention aligned with the `kubeflow_trainer_` namespace and
Kubernetes instrumentation SIG guidelines
- Deciding scope: controller-health metrics vs. TrainJob-lifecycle metrics vs. plugin-chain metrics
- ServiceMonitor needs to work with the existing TLS-secured `:8443/metrics` endpoint
- Peer projects (Kueue, JobSet, Volcano) all ship custom metrics - Trainer is a gap

### Why is this needed?

### Current State

The controller relies solely on default controller-runtime metrics. There are:

- No application-defined metrics - no custom counters, histograms, or gauges in trainer code
- No `ServiceMonitor` or `PodMonitor` in Helm or kustomize - default metrics aren't easily
scrapeable by Prometheus Operator
- No visibility into TrainJob lifecycle (time-to-running, completion rates, failure rates)
from the operator's perspective

This was flagged in 2021 ([#1340](https://github.com/kubeflow/trainer/issues/1340),
[#1318](https://github.com/kubeflow/trainer/issues/1318)) but never delivered for Trainer v2.

### Related Issues

- https://github.com/kubeflow/trainer/issues/2779
- https://github.com/kubeflow/trainer/issues/2648
- https://github.com/kubeflow/trainer/issues/1340
- https://github.com/kubeflow/trainer/issues/1318

### Love this feature?

Give it a 👍 We prioritize the features with most 👍