lablup / lablup/backend.ai

Rewrite Idle Checker on Prometheus

Open
#11,817 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Background

The current Idle Checker (`src/ai/backend/manager/idle.py`, ~1,347 lines) maintains all per-session timing and utilization state in Valkey/Redis:
- `session.{sid}.last_access` — network-activity timestamp
- `session.{sid}.util_series` / `util_first_collected` / `util_last_collected` — utilization time-window samples
- `session.{sid}.utilization_extra` / `\*.report` — checker output cache
- Stat DB `kernel.{kid}` — per-kernel utilization scraped from agents

Each 15-second tick (`GlobalTimer` + `DoIdleCheckEvent`) reads/writes these keys for every LIVE session, coupling manager idle decisions tightly to Valkey availability and consistency.

Meanwhile, agent-side resource metrics are already (or can be) exported to Prometheus, which provides native time-window aggregation (rate/avg over interval) without manager-side bookkeeping.

## Goal

Replace the Valkey-backed state of the idle checker with Prometheus-based queries:
- Utilization windows → PromQL range queries against agent-exported metrics
- Network activity timestamps → Prometheus gauge updated on session/exec/streaming events (or alternative push path)
- Keep only termination-event coordination in the event bus; remove Valkey from the hot evaluation path

Preserve existing checker semantics (NetworkTimeout, Utilization, SessionLifetime, NewUserGracePeriod) and the public report API (`get_idle_check_report`, GQL/REST surface).

## Scope

- Manager: `src/ai/backend/manager/idle.py`, `event_dispatcher/handlers/idle_check.py`, `dependencies/orchestration/idle_checker.py`
- Agent: any new Prometheus metric exports required for last-access timestamps and per-resource utilization
- Config: TOML/etcd idle config schema migration if new knobs needed (PromQL endpoint, query templates)
- Tests: unit tests for new query/aggregation logic, integration test against a Prometheus instance

## Out of Scope

- Changing the user-facing termination policy or grace-period behavior
- Replacing Valkey for unrelated subsystems

## Constraints

- Backward-compatible report API for Web UI / Client SDK
- Must work without Prometheus being highly available (define degradation behavior)
- Per-keypair `idle_timeout` from `keypair_resource_policies` must still be honored

## Related

- Supersedes BA-1138 (utilization checker → Prometheus, Story-sized — closed as Not Planned in favor of this Epic-level rewrite)
- Adjacent: BA-5314, BA-2676, BA-2551, BA-85

JIRA Issue: BA-6197

Contributor guide

Open the contributing guide

Research direction

Start by reading src/ai/backend/manager/idle.py, event_dispatcher/handlers/idle_check.py, and dependencies/orchestration/idle_checker.py, then inspect the agent metric-export paths. Define the required PromQL queries and degradation behavior before changing the checker. Done means existing idle semantics and report APIs remain compatible, with unit coverage and an integration test against Prometheus.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
api, backend, distributed-systems, observability-sre, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.