lablup / lablup/backend.ai

Migrate resource_usage get_user_kernel_statistics_batch callers off Valkey

Open
#11,703 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Background

Three user/group resource-usage queries still read live kernel statistics from Valkey via ValkeyStatClient.get_user_kernel_statistics_batch. With kernel metrics migrating to Prometheus, these callers must be cut over so we can finally drop ValkeyStatClient and its kernel-stat keys (tracked by BA-4051).

## Target call sites
- src/ai/backend/manager/models/resource_usage.py:504 — get_user_kernel_statistics_batch in fetch_resource_usage helper
- src/ai/backend/manager/repositories/group/db_source/db_source.py:409 — group resource-usage query
- src/ai/backend/manager/repositories/user/repository.py:439 — user resource-usage query

All three follow the same pattern: collect kernel_ids → fetch live stats from Valkey → merge into the result. Pick one of two strategies (both acceptable):
1. DB-only path: drop the Valkey lookup and rely on kernels.last_stat already persisted in PostgreSQL.
2. Prometheus path: query ContainerMetricService / MetricRepository for the same fields and merge.

Decide the strategy up front and apply it consistently across all three call sites.

## Scope
- Replace each get_user_kernel_statistics_batch call with the chosen alternative.
- Remove now-unused imports of ValkeyStatClient / valkey_stat_client from the three files.
- Update any helper functions in models/resource_usage.py that exist only to merge the Valkey payload.

## Success Criteria
- [ ] None of the three files reference get_user_kernel_statistics_batch
- [ ] Resource-usage GraphQL queries for user/group return cpu/memory/io fields equivalent to current behavior (verified manually against a running session)
- [ ] grep -rn "get_user_kernel_statistics_batch" src/ai/backend/manager/ returns only the ValkeyStatClient definition (no callers)
- [ ] pants check passes for manager package
- [ ] pants test passes for affected packages

JIRA Issue: BA-6114

Contributor guide

Open the contributing guide

Research direction

Start by reading the three named call sites in resource_usage.py, group/db_source.py, and user/repository.py, then inspect ContainerMetricService, MetricRepository, and kernels.last_stat to choose the DB-only or Prometheus strategy. Apply that choice consistently, remove obsolete helpers and imports, and verify with grep, the affected GraphQL resource-usage queries, pants check, and pants test.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, postgresql, prometheus, python
Domain
api, backend, databases, observability
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.