lablup / lablup/backend.ai

Make agent utilization metric interval configurable and derive Valkey TTL from it

Open
#12,157 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Summary

The agent collects utilization metrics on a fixed interval that is currently hardcoded, and the collected values are stored in Valkey with hardcoded expirations. This makes it impossible to tune the collection cadence per deployment, and the Valkey TTLs do not adapt when the interval changes. Expose the utilization metric interval as a configuration value, and derive the Valkey expiration of utilization values from that interval so they stay consistent.

## Current State

- Collection interval is a hardcoded constant UTILIZATION_METRIC_INTERVAL = 5.0 in src/ai/backend/common/metrics/types.py, used by the agent stat-collection timers (agent.py, runtime.py) and to derive the stats plugin timeout (stats.py).
- Utilization values are written to Valkey with hardcoded/independent expirations: agent-level metrics use cache_lifespan (120s), container/process metrics use a hardcoded expire_sec=8 (stats.py). These TTLs are unrelated to the collection interval.

## Goal

- Add a configuration option for the utilization metric collection interval in the agent unified config schema, with the current 5.0s as the default.
- Wire the configured interval through the agent stat-collection timers and any derived values (e.g. the stats plugin timeout).
- Derive the Valkey expiration of utilization values from the configured interval (e.g. a multiple of the interval) instead of hardcoded TTLs, so the expiration scales when the interval is changed.
- Preserve current effective behavior when the config is left at its default.

## Acceptance Criteria

- The utilization metric interval is configurable via agent config and defaults to the existing value.
- Valkey expiration for stored utilization values is computed from the configured interval, not a hardcoded number.
- Changing the interval in config visibly changes both the collection cadence and the Valkey TTLs.

JIRA Issue: BA-6471

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.