Rewrite Idle Checker on Prometheus
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Background
The current Idle Checker (`src/ai/backend/manager/idle.py`, ~1,347 lines) maintains all per-session timing and utilization state in Valkey/Redis:
- `session.{sid}.last_access` — network-activity timestamp
- `session.{sid}.util_series` / `util_first_collected` / `util_last_collected` — utilization time-window samples
- `session.{sid}.utilization_extra` / `\*.report` — checker output cache
- Stat DB `kernel.{kid}` — per-kernel utilization scraped from agents
Each 15-second tick (`GlobalTimer` + `DoIdleCheckEvent`) reads/writes these keys for every LIVE session, coupling manager idle decisions tightly to Valkey availability and consistency.
Meanwhile, agent-side resource metrics are already (or can be) exported to Prometheus, which provides native time-window aggregation (rate/avg over interval) without manager-side bookkeeping.
## Goal
Replace the Valkey-backed state of the idle checker with Prometheus-based queries:
- Utilization windows → PromQL range queries against agent-exported metrics
- Network activity timestamps → Prometheus gauge updated on session/exec/streaming events (or alternative push path)
- Keep only termination-event coordination in the event bus; remove Valkey from the hot evaluation path
Preserve existing checker semantics (NetworkTimeout, Utilization, SessionLifetime, NewUserGracePeriod) and the public report API (`get_idle_check_report`, GQL/REST surface).
## Scope
- Manager: `src/ai/backend/manager/idle.py`, `event_dispatcher/handlers/idle_check.py`, `dependencies/orchestration/idle_checker.py`
- Agent: any new Prometheus metric exports required for last-access timestamps and per-resource utilization
- Config: TOML/etcd idle config schema migration if new knobs needed (PromQL endpoint, query templates)
- Tests: unit tests for new query/aggregation logic, integration test against a Prometheus instance
## Out of Scope
- Changing the user-facing termination policy or grace-period behavior
- Replacing Valkey for unrelated subsystems
## Constraints
- Backward-compatible report API for Web UI / Client SDK
- Must work without Prometheus being highly available (define degradation behavior)
- Per-keypair `idle_timeout` from `keypair_resource_policies` must still be honored
## Related
- Supersedes BA-1138 (utilization checker → Prometheus, Story-sized — closed as Not Planned in favor of this Epic-level rewrite)
- Adjacent: BA-5314, BA-2676, BA-2551, BA-85
JIRA Issue: BA-6197
Contributor guide
Research direction
Start by reading src/ai/backend/manager/idle.py, event_dispatcher/handlers/idle_check.py, and dependencies/orchestration/idle_checker.py, then inspect the agent metric-export paths. Define the required PromQL queries and degradation behavior before changing the checker. Done means existing idle semantics and report APIs remain compatible, with unit coverage and an integration test against Prometheus.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, python
- Domain
- api, backend, distributed-systems, observability-sre, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100