lablup / lablup/backend.ai

Refactor Agent Kernel Metrics Cleanup with Sliding Window Observer

Open
#7,535 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Problem

Timing hole in kernel metrics cleanup causes metric data removal failures.

**Current flow:**

1. Container removed → `collect_container_stat()` removes kernel from `self.kernel_metrics` (stats.py:658-661)
1. `remove_kernel_metric()` called later (agent.py:1505, 1641)
1. Kernel info already gone → metric cleanup fails

**Root cause:** Direct removal pattern is timing-dependent.

## Solution

Replace explicit removal with sliding window observer:

- Auto-cleanup metrics not collected for configurable period (multiple of 5min)
- Remove direct removal patterns from codebase
- Time-based, predictable metric retention

## Scope

- `src/ai/backend/agent/stats.py`
- `src/ai/backend/agent/agent.py`
- Configuration for cleanup window

JIRA Issue: BA-3521

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.