Scaling Group mutation is not propagated to Agent local config
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Summary
When an administrator updates an agent's scaling group via the `update_scaling_group()` RPC call, the TOML configuration file is updated correctly and agent restart works as expected. However, the runtime `local_config` update fails - only `AgentRPCServer.local_config` reflects the new value while `AbstractAgent.local_config` continues to hold the old value. This causes the agent to continue reporting the old scaling group in heartbeats until restart.
## Steps to Reproduce
1. Deploy [http://Backend.AI](http://Backend.AI) with tag version 25.15.5
1. Start an agent with scaling group "default"
1. Call `update_scaling_group("new-group")` RPC method from manager
1. Check `AgentRPCServer.local_config.agent.scaling_group` → Shows "new-group" ✓
1. Check agent heartbeat logs or `AbstractAgent.local_config.agent.scaling_group` → Shows "default" ✗
1. Observe that agent continues reporting "default" to manager in heartbeats
1. Restart the agent → Both now show "new-group" (proving TOML was updated correctly)
## Expected Behavior
Both `AgentRPCServer.local_config` and `AbstractAgent.local_config` should reference the same `AgentUnifiedConfig` object. When `update_scaling_group()` updates the scaling group, both should immediately reflect the new value without requiring an agent restart.
## Actual Behavior
Only `AgentRPCServer.local_config` is updated with the new scaling group. `AbstractAgent.local_config` retains the old value because the shared object reference is broken during the update process. The agent continues to report the old scaling group to the manager in heartbeat messages.
## Root Cause
**Location**: `src/ai/backend/agent/server.py:613` (tag version 25.15.5)
**Problematic code**:
```python
def update_scaling_group(self, request: web.Request) -> web.Response:
scaling_group = await request.json()
new_agent_config = self.local_config.agent.model_copy(
update={"scaling_group": scaling_group}
)
# BUG: This creates a NEW AgentUnifiedConfig object
self.local_config = self.local_config.model_copy(update={"agent": new_agent_config})
```
**Technical explanation:**
1. Initial state: Both AgentRPCServer and AbstractAgent reference the same AgentUnifiedConfig object (Object A)
- In single-agent mode: get_agent_configs() returns [self]
- Both share the same config: server.local_config is agent.local_config → True
1. After update: model_copy() creates a new object (Object B)
- AgentRPCServer.local_config → Object B (new scaling group)
- AbstractAgent.local_config → Object A (old scaling group)
- Shared reference broken: server.local_config is agent.local_config → False
1. Consequence: Agent heartbeat reads self.local_config.agent.scaling_group which still points to Object A with the old value
## Solution
```python
def update_scaling_group(self, request: web.Request) -> web.Response:
scaling_group = await request.json()
new_agent_config = self.local_config.agent.model_copy(
update={"scaling_group": scaling_group}
)
# FIX: Only replace the nested agent field, not the entire config object
self.local_config.agent = new_agent_config
```
## Testing
The test verifies:
- Initial state: server.local_config and agent.local_config share the same reference
- After update: Config object reference is preserved (id(config) remains unchanged)
- After update: Both server and agent see the new scaling group value
- TOML file is updated correctly for restart scenarios
Test design:
- Server-level integration test with 9 separated fixtures
- Mocks: AsyncEtcd, AgentStatsPluginContext, AgentErrorPluginContext, RPC components
- Uses shortened temp path to avoid AF_UNIX socket path length issues
- All setup in fixtures; test method contains only test logic
JIRA Issue: BA-3560
Contributor guide
Assessment
This issue has not been assessed yet.