Prometheus metrics for operator and key-manager

Problem

Today the only way to know what the pack is doing is to read pod logs. There are no metrics, no dashboards, no SLO targets, and no alerting hooks. For a production deployment this means:

• No visibility into reconcile latency, error rates, or queue depth on the operator.
• No visibility into key-manager request rates, error rates, or which model groups are hot.
• No way to tell internal server error rates apart by handler / by model.
• No way to detect a stuck reconciler or a key-manager pod that has lost its k8s watch.

Proposal

Operator

Use the controller-runtime metrics endpoint that's already wired up by kubebuilder. Expose:

• controller_runtime_reconcile_total{controller, result} (built-in)
• controller_runtime_reconcile_errors_total{controller} (built-in)
• controller_runtime_reconcile_time_seconds{controller} histogram (built-in)
• Custom: nebari_llmmodel_total{phase} gauge (one per LLMModel by phase: Pending / Ready / Failed)
• Custom: nebari_llmmodel_reconcile_failures_total{model_namespace, model_name, reason} counter

The controller-runtime metrics are served at :8080/metrics by default - just confirm the chart enables it and exposes it via a Service that a ServiceMonitor (or PodMonitor) can scrape.

Key-manager

Add a /metrics endpoint on the existing HTTP server. Expose:

• keymanager_http_requests_total{handler, method, status} counter
• keymanager_http_request_duration_seconds{handler, method} histogram
• keymanager_keys_total{model} gauge (live count from the k8s cache)
• keymanager_key_operations_total{operation, result} counter (operation in {create, list, revoke}; result in {success, error})
• keymanager_k8s_watch_errors_total{resource} counter (informer watch failures)

Use prometheus/client_golang and the promhttp.Handler(). Wire structured-log error counts up to a metric so the #57 logging fix has a metric companion.

Charts

Both subcharts should ship a ServiceMonitor template gated on serviceMonitor.enabled (off by default), pointing at the metrics endpoint. Document the assumed Prometheus Operator availability.

Out of scope

• Dashboards (Grafana JSON) - separate issue once metrics exist.
• Tracing (OTLP) - separate issue.
• SLOs / alerts - separate issue.

This issue is just "metrics exist and a Prometheus can scrape them".

Locate the existing controller-runtime metrics wiring, key-manager HTTP server, Kubernetes cache, and both Helm subcharts; start by tracing how endpoints and Service resources are exposed. Done means the specified operator and key-manager metrics are registered, both optional ServiceMonitor templates scrape them, and the Prometheus Operator assumption is documented.