container_utilization_metric returns cpu_util as percent × 10: exporter publishes the raw msec counter with capacity 1000 and never pct

## Symptom

`user_utilization_metric` / `container_utilization_metric` (GraphQL type `ContainerUtilizationMetric`) return `cpu_util` in **percent × 10**: a 1-core container at full load reports `values[].value` ≈ 1000 and `capacity` = 1000, while `avg_value` / `max_value` are in the same scale. Every other `*_util` metric (e.g. `cuda_util`) is plain percent (100 = 100 %). Observed on `frontend.latest.staging.backend.ai` on 2026-09-15 (raw 969 for a busy-loop on one core).

The WebUI currently hides this with a `cpu_util`-only `/ 10` in `SessionMetricGraph.tsx`, and that special case is what let the Avg Used reference line drift to 10× in FR-3932 (lablup/backend.ai-webui#9663).

## Cause

The agent exports the raw cumulative CPU-time counter and never publishes `pct` on the container path; no later layer divides by capacity. Verified on `main` @ 8545de4841 (2026-09-11):

1. **Measurement** — `src/ai/backend/agent/docker/intrinsic.py:350-353`: `cpu_util` is `Measurement(cpu_used_msec, capacity=Decimal(1000))`. The value is cumulative CPU time in **msec**; capacity is the constant 1000 (msec/s = one core), independent of the container's core count. Line 358 still declares `unit_hint="percent"`.
1. **Exporter** — `src/ai/backend/agent/stats.py:679-702` (`observe_container_metric`) sets the `backendai_container_utilization` gauge with `value_type="current"` ← `measure.value` (the msec counter) and `value_type="capacity"` ← 1000. No `pct` series. The `kernel_updates` list built at `stats.py:815-847` **does** include `PCT_METRIC_KEY`, but it is only written to a debug log at line 849 and never handed to the observer — dead code.
1. **PromQL** — `src/ai/backend/manager/clients/prometheus/metric_types.py:77` lists `cpu_util` in `DIFF_METRICS`, so `fixed_query_builder.py:29-33, 84-90` wraps `current` in `rate(...)`. `rate()` over a msec counter yields msec per second → **1000 per fully loaded core**. `capacity` is not DIFF and stays a plain gauge → 1000.
1. **GraphQL** — `src/ai/backend/manager/api/gql_legacy/metric/base.py:57-67` computes `avg_value` / `max_value` as the plain mean / max of those samples. No scaling and no unit field.

## Why other metrics are fine

- `cuda_util` (`src/ai/backend/accelerator/cuda_open/plugin.py:577-591`): the NVML utilization is already a percent and capacity is `devices × 100`; it is neither DIFF nor RATE, so the manager returns it unchanged. Same shape in the rocm / ipu / habana / mock plugins.
- The legacy `live_stat` path recomputes `pct = current / capacity × 100` in `src/ai/backend/manager/api/gql_legacy/stat_converter.py:136-139`, and `KernelNode.cpu_util` reads that `pct`. Only the newer `ContainerUtilizationMetric` path skips the normalization.

## Side findings

- Because capacity is a constant 1000 (one core), a 4-core container at full load reports `current` 4000 against `capacity` 1000. `cpu_util` is core-based and can exceed 100 %, whereas accelerator `*_util` metrics are capped at 100 % — so the two need different formulas, and the client currently has no way to know which one applies.
- The two places that state a unit for `cpu_util` (`intrinsic.py:358`, `stat_converter.py:21`) both say `percent`, which does not describe the emitted series (msec per second, 1000 per core). No docs mention msec / millicores.
- The dummy backend (`src/ai/backend/agent/dummy/compute_backend.py:32`) gives `cpu_util` a capacity of 100, a third convention.

## Agreed direction (Teams thread, 2026-09-15)

1. **Serve `pct` from the server** for container utilization metrics, computed as `current / capacity × 100` the way the legacy `live_stat` converter already does (`stat_converter.py:136-139`). For `cpu_util` this is percent of one core and may exceed 100 % on multi-core containers; for accelerator metrics it stays within 0–100 %.
1. **Add a unit hint later** so the WebUI can label the value (for example, mark `cpu_util` as "per core, may exceed 100 %") instead of assuming every `*_util` is a 0–100 % gauge.

Once `pct` is available, the WebUI drops its `cpu_util / 10` in `react/src/components/SessionMetricGraph.tsx` (`convertMetricUnit`) and reads `pct` for every `*_util` metric.

## Related

- FR-3932 — WebUI symptom (Avg Used line at 10×), fixed on the frontend by lablup/backend.ai-webui#9663.
- Teams discussion (devops / JIRA Issue Notification, 2026-09-15) — attached as a web link on this issue.

JIRA Issue: BA-7900

Start with observe_container_metric in src/ai/backend/agent/stats.py and compare its handling with stat_converter.py:136-139; then trace the metric through metric_types.py and fixed_query_builder.py. Check the ContainerUtilizationMetric fields in base.py and the consumer in react/src/components/SessionMetricGraph.tsx. Done means the server exposes normalized pct values for utilization metrics and the client no longer applies the cpu_util-only scaling.

Written by the indexing model from the issue text.