lablup / lablup/backend.ai

Serve pct for container utilization metrics from the manager (value_type: "pct")

Open
#14,641 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Goal

`user_utilization_metric` (graphene `ContainerUtilizationMetric`) accepts `value_type: "pct"` and returns `current / capacity × 100` computed by the manager in PromQL, for every container metric. The WebUI can then drop its cpu_util-only `/ 10` and read pct for all \*_util panels, including the Avg Used reference line.

## Background

- The agent publishes only `current` and `capacity` to `backendai_container_utilization` (agent/stats.py observe_container_metric). For cpu_util, current is the cumulative CPU-time msec counter and capacity is 1000 (msec/s, one core), so the manager's rate() output is percent × 10 (BA-7900, FR-3932).
- `ValueType.PCT` exists in manager/clients/prometheus/types.py but nothing references it; a "pct" request is not rejected and renders a gauge selector on a series that does not exist, so it always returns an empty list.
- Agreed direction (Teams thread, 2026-09-15, recorded on BA-7900): serve pct from the server the way the legacy live_stat converter already does (api/gql_legacy/stat_converter.py: rate_current or instant_current, divided by capacity when capacity > 0). A unit hint (cpu_util is per core and may exceed 100 %) is a later follow-up.

## Deliverable

- `clients/prometheus/fixed_query_builder.py`: pct templates — `label_replace(sum by (g)(current) / (sum by (g)(capacity) > 0) * 100, "value_type", "pct", "", "")`; the numerator is wrapped in rate() for the cumulative-counter metrics (DIFF_METRICS ∪ RATE_METRICS: cpu_util, net_rx, net_tx), the same set the live_stat rate queries use.
- `clients/prometheus/querier.py`: for pct, omit `value_type` from the label matchers and from group_by so both sides of the division match on the same label set; the template pins value_type on each side.
- GraphQL descriptions of `value_type` (UserUtilizationMetricQueryInput, ContainerUtilizationMetric) list 'pct'; schema dumps updated.
- Unit tests for the rendered PromQL (cpu_util / net_rx / cuda_util) and for the querier label set.

## Acceptance criteria

- `value_type: "current"` and `"capacity"` render exactly the same PromQL as before.
- `value_type: "pct"` for cpu_util returns rate(current) / capacity × 100 (1 core fully used → 100; per-kernel queries equal the legacy live_stat cpu_util.pct); for gauge util metrics (cuda_util, …) returns current / capacity × 100 within 0–100.
- A user-scoped pct query aggregates as Σcurrent / Σcapacity over the user's kernels.
- A zero or missing capacity yields no series (no +Inf / NaN reaches avg_value / max_value).
- Every returned series carries `value_type="pct"` so ContainerMetricResponseInfo parsing and the GQL value_type field are unchanged.
- Works on data already stored in Prometheus; no agent upgrade required.

## Out of scope

- Agent changes: exporting pct to Prometheus, or scaling cpu_util capacity by allocated cores (BA-7872).
- Unit hint for per-core metrics.
- WebUI switch to pct (backend.ai-webui).

## Related

- [BA-7900 — symptom report and agreed direction](https://lablup.atlassian.net/browse/BA-7900)
- [FR-3932 — WebUI Avg Used line at 10×](https://lablup.atlassian.net/browse/FR-3932)
- [BA-7872 — cpu_util capacity fixed at one core (agent)](https://lablup.atlassian.net/browse/BA-7872)
- [lablup/backend.ai-webui#9663 — frontend fix for FR-3932](https://github.com/lablup/backend.ai-webui/pull/9663)

JIRA Issue: BA-7906

Contributor guide

Open the contributing guide

Research direction

Read clients/prometheus/fixed_query_builder.py and clients/prometheus/querier.py, using api/gql_legacy/stat_converter.py's rate_current/instant_current behavior as the reference; then inspect the GraphQL descriptions and schema dumps. Add the pct query coverage for cpu_util, net_rx, and cuda_util plus querier label-set tests, and verify current/capacity queries remain unchanged and pct results omit zero-capacity series.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, prometheus, python
Domain
api, backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.