Fix agent scaling group update with active kernels
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Problem
When `update_scaling_group()` is called on an agent with running kernels, only the agent's `scaling_group` field is updated while the kernels' `scaling_group` values remain unchanged. This causes the Sokovan scheduler to incorrectly calculate available agent resources, leading to over-allocation and scheduling failures.
## Root Cause
1. **Agent update**: Only `agents.scaling_group` is updated via ModifyAgent mutation
1. **Kernel state**: Running kernels on the agent retain their old `scaling_group` value
1. **Scheduler query**: Sokovan filters kernels by `KernelRow.scaling_group == scaling_group` when calculating occupancy
1. **Result**: Orphaned kernels are excluded from resource calculations, causing incorrect availability
## Impact
- **Severity**: High
- **Symptoms**: Resource over-allocation, kernel creation failures, inconsistent state
## Solution Approach
Block scaling_group updates when agent has active kernels. This is a conservative fix that prevents data inconsistency.
## Sub-tasks
1. Add validation logic to prevent scaling_group update when active kernels exist
1. Refactor update_scaling_group() logic for better error handling and clarity
## References
- Key files:
- `src/ai/backend/manager/models/gql_models/agent.py:888-891` (ModifyAgent mutation)
- `src/ai/backend/manager/repositories/schedule/db_source/db_source.py:309` (Scheduler query)
- `src/ai/backend/manager/models/kernel.py:405-407` (Kernel.scaling_group field)
JIRA Issue: BA-3569
Contributor guide
Assessment
This issue has not been assessed yet.