manager/agent Prometheus multi-process metrics silently drop all samples due to early prometheus_client import
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Symptom
- `curl http://127.0.0.1:6003/metrics` (agent) and `curl http://127.0.0.1:18080/metrics` (manager) return `Content-Length: 0` even while a kernel container is actively running.
- `run/prometheus/agent/` and `run/prometheus/manager/` contain no `.db` files (storage component is unaffected and has its files).
- As a downstream effect, Prometheus never scrapes any `backendai_container_utilization` samples, which causes:
- Auto-scaling rules attached to an endpoint to see no metric data.
- `prometheusQueryPresetResult(...)` GQL query to always return `result: []`.
## Root Cause
`src/ai/backend/common/metrics/multiprocess.py` imports `prometheus_client` at **module top level**:
```python
from prometheus_client import CollectorRegistry, generate_latest
from prometheus_client.multiprocess import MultiProcessCollector
from prometheus_client.multiprocess import mark_process_dead as _mark_dead
```
`prometheus_client.values.py:139` executes `ValueClass = get_value_class()` exactly once at import time, branching on the current value of `PROMETHEUS_MULTIPROC_DIR`.
CLI entry points (`agent/cli/start_server.py`, `manager/cli/start_server.py`) perform this sequence inside `main()`:
1. `from ai.backend.common.metrics.multiprocess import setup_prometheus_multiprocess_dir` — triggers `prometheus_client.values` to evaluate `get_value_class()` while env is still unset → `ValueClass = MutexValue` (single-process).
1. `setup_prometheus_multiprocess_dir("agent")` — sets `PROMETHEUS_MULTIPROC_DIR`, but `ValueClass` is already bound and is never re-evaluated.
1. Worker processes are later spawned via `aiotools.start_server` using the default fork `mp_context`. Children inherit the parent's `ValueClass = MutexValue`.
Result: metric `.set()` calls go through `MutexValue.set()` which only mutates in-process memory, producing no `.db` files. `/metrics` served via `MultiProcessCollector` reads from `PROMETHEUS_MULTIPROC_DIR` — empty → empty HTTP response.
### Why Storage Is Not Affected
`src/ai/backend/storage/server.py:818` contains `multiprocessing.set_start_method("spawn")`. Spawned workers start a fresh Python interpreter, inherit `PROMETHEUS_MULTIPROC_DIR` as an OS env var, and re-import `prometheus_client` correctly — binding `ValueClass = MmapedValue`. Agent and manager do not call `set_start_method("spawn")` and therefore hit the bug.
## Reproduction
```python
import os, sys
from pathlib import Path
from ai.backend.common.metrics.multiprocess import setup_prometheus_multiprocess_dir
# multiprocess.py already pulled in prometheus_client here → ValueClass = MutexValue
setup_prometheus_multiprocess_dir("repro", base_dir=Path("/tmp/r"))
from prometheus_client import values, Gauge
print(values.ValueClass.__name__) # MutexValue
Gauge("t", "d", ["a"], multiprocess_mode="livesum").labels(a="x").set(1.0)
print(os.listdir("/tmp/r/repro")) # []
```
## Fix
Move the `prometheus_client` imports in `common/metrics/multiprocess.py` from module top level into the function bodies that actually use them (`generate_latest_multiprocess`, `generate_latest_singleprocess`, `mark_process_dead`). With this change, merely importing `setup_prometheus_multiprocess_dir` no longer forces `prometheus_client` to be imported, so the env var set by `setup_prometheus_multiprocess_dir()` is in place by the time any downstream `from prometheus_client import ...` runs.
Verified end-to-end:
- `run/prometheus/agent/` and `run/prometheus/manager/` now contain the three `.db` files per worker pid.
- `curl :6003/metrics` returns ~40 KB with 12 `backendai_container_utilization` samples.
- `curl :18080/metrics` returns ~74 KB with 504 `backendai_*` metrics.
- Prometheus successfully scrapes the samples.
- `prometheusQueryPresetResult` GQL query returns non-empty results.
## Affected Versions
Present on main at commit `ca5331f52` (2026-04-14). Likely affects every release that has `src/ai/backend/common/metrics/multiprocess.py` with top-level `prometheus_client` imports.
## Scope of Impact
- **Manager**: `/metrics` silent data loss — Grafana dashboards backed by manager metrics (request latency, GraphQL counters, layer operation counts, etc.) receive nothing.
- **Agent**: `backendai_container_utilization`, `backendai_device_utilization`, `backendai_process_utilization` all silently dropped. Auto-scaling rules backed by Prometheus metrics never fire.
- **Account-manager, app-proxy coordinator/worker**: same failure mode — they use the same `setup_prometheus_multiprocess_dir` pattern without `set_start_method("spawn")`. (Not verified directly; recommended to check.)
## Acceptance Criteria
- With fresh local dev stack, `run/prometheus/{manager,agent,appproxy-coordinator,appproxy-worker,account-manager}/` each contain `.db` files per worker pid.
- `curl :6003/metrics` and `curl :18080/metrics` return non-empty responses containing `backendai_*` series.
- Prometheus has non-zero samples for `backendai_container_utilization` while any kernel is running.
- `prometheusQueryPresetResult` GQL returns non-empty result when a healthy replica is running.
- `pants fmt|lint|check|test` pass for touched files.
## Note
Storage is the only component that avoids the bug, accidentally, via `multiprocessing.set_start_method("spawn")`. That line can optionally stay or be removed once this fix lands — it is no longer needed for multi-process metrics to work.
—
Captured while working on branch: main
JIRA Issue: BA-5707
Contributor guide
Assessment
This issue has not been assessed yet.