lablup / lablup/backend.ai

Fix agent scaling group update with active kernels

Open
#7,595 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Problem

When `update_scaling_group()` is called on an agent with running kernels, only the agent's `scaling_group` field is updated while the kernels' `scaling_group` values remain unchanged. This causes the Sokovan scheduler to incorrectly calculate available agent resources, leading to over-allocation and scheduling failures.

## Root Cause

1. **Agent update**: Only `agents.scaling_group` is updated via ModifyAgent mutation
1. **Kernel state**: Running kernels on the agent retain their old `scaling_group` value
1. **Scheduler query**: Sokovan filters kernels by `KernelRow.scaling_group == scaling_group` when calculating occupancy
1. **Result**: Orphaned kernels are excluded from resource calculations, causing incorrect availability

## Impact

- **Severity**: High
- **Symptoms**: Resource over-allocation, kernel creation failures, inconsistent state

## Solution Approach

Block scaling_group updates when agent has active kernels. This is a conservative fix that prevents data inconsistency.

## Sub-tasks

1. Add validation logic to prevent scaling_group update when active kernels exist
1. Refactor update_scaling_group() logic for better error handling and clarity

## References

- Key files:
- `src/ai/backend/manager/models/gql_models/agent.py:888-891` (ModifyAgent mutation)
- `src/ai/backend/manager/repositories/schedule/db_source/db_source.py:309` (Scheduler query)
- `src/ai/backend/manager/models/kernel.py:405-407` (Kernel.scaling_group field)

JIRA Issue: BA-3569

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.