Serve pct for container utilization metrics from the manager (value_type: "pct")
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Goal
`user_utilization_metric` (graphene `ContainerUtilizationMetric`) accepts `value_type: "pct"` and returns `current / capacity × 100` computed by the manager in PromQL, for every container metric. The WebUI can then drop its cpu_util-only `/ 10` and read pct for all \*_util panels, including the Avg Used reference line.
## Background
- The agent publishes only `current` and `capacity` to `backendai_container_utilization` (agent/stats.py observe_container_metric). For cpu_util, current is the cumulative CPU-time msec counter and capacity is 1000 (msec/s, one core), so the manager's rate() output is percent × 10 (BA-7900, FR-3932).
- `ValueType.PCT` exists in manager/clients/prometheus/types.py but nothing references it; a "pct" request is not rejected and renders a gauge selector on a series that does not exist, so it always returns an empty list.
- Agreed direction (Teams thread, 2026-09-15, recorded on BA-7900): serve pct from the server the way the legacy live_stat converter already does (api/gql_legacy/stat_converter.py: rate_current or instant_current, divided by capacity when capacity > 0). A unit hint (cpu_util is per core and may exceed 100 %) is a later follow-up.
## Deliverable
- `clients/prometheus/fixed_query_builder.py`: pct templates — `label_replace(sum by (g)(current) / (sum by (g)(capacity) > 0) * 100, "value_type", "pct", "", "")`; the numerator is wrapped in rate() for the cumulative-counter metrics (DIFF_METRICS ∪ RATE_METRICS: cpu_util, net_rx, net_tx), the same set the live_stat rate queries use.
- `clients/prometheus/querier.py`: for pct, omit `value_type` from the label matchers and from group_by so both sides of the division match on the same label set; the template pins value_type on each side.
- GraphQL descriptions of `value_type` (UserUtilizationMetricQueryInput, ContainerUtilizationMetric) list 'pct'; schema dumps updated.
- Unit tests for the rendered PromQL (cpu_util / net_rx / cuda_util) and for the querier label set.
## Acceptance criteria
- `value_type: "current"` and `"capacity"` render exactly the same PromQL as before.
- `value_type: "pct"` for cpu_util returns rate(current) / capacity × 100 (1 core fully used → 100; per-kernel queries equal the legacy live_stat cpu_util.pct); for gauge util metrics (cuda_util, …) returns current / capacity × 100 within 0–100.
- A user-scoped pct query aggregates as Σcurrent / Σcapacity over the user's kernels.
- A zero or missing capacity yields no series (no +Inf / NaN reaches avg_value / max_value).
- Every returned series carries `value_type="pct"` so ContainerMetricResponseInfo parsing and the GQL value_type field are unchanged.
- Works on data already stored in Prometheus; no agent upgrade required.
## Out of scope
- Agent changes: exporting pct to Prometheus, or scaling cpu_util capacity by allocated cores (BA-7872).
- Unit hint for per-core metrics.
- WebUI switch to pct (backend.ai-webui).
## Related
- [BA-7900 — symptom report and agreed direction](https://lablup.atlassian.net/browse/BA-7900)
- [FR-3932 — WebUI Avg Used line at 10×](https://lablup.atlassian.net/browse/FR-3932)
- [BA-7872 — cpu_util capacity fixed at one core (agent)](https://lablup.atlassian.net/browse/BA-7872)
- [lablup/backend.ai-webui#9663 — frontend fix for FR-3932](https://github.com/lablup/backend.ai-webui/pull/9663)
JIRA Issue: BA-7906
Contributor guide
Research direction
Read clients/prometheus/fixed_query_builder.py and clients/prometheus/querier.py, using api/gql_legacy/stat_converter.py's rate_current/instant_current behavior as the reference; then inspect the GraphQL descriptions and schema dumps. Add the pct query coverage for cpu_util, net_rx, and cuda_util plus querier label-set tests, and verify current/capacity queries remain unchanged and pct results omit zero-capacity series.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- graphql, prometheus, python
- Domain
- api, backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100