lablup / lablup/backend.ai

Scaling Group mutation is not propagated to Agent local config

Open
#7,585 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Summary

When an administrator updates an agent's scaling group via the `update_scaling_group()` RPC call, the TOML configuration file is updated correctly and agent restart works as expected. However, the runtime `local_config` update fails - only `AgentRPCServer.local_config` reflects the new value while `AbstractAgent.local_config` continues to hold the old value. This causes the agent to continue reporting the old scaling group in heartbeats until restart.

## Steps to Reproduce

1. Deploy [http://Backend.AI](http://Backend.AI) with tag version 25.15.5
1. Start an agent with scaling group "default"
1. Call `update_scaling_group("new-group")` RPC method from manager
1. Check `AgentRPCServer.local_config.agent.scaling_group` → Shows "new-group" ✓
1. Check agent heartbeat logs or `AbstractAgent.local_config.agent.scaling_group` → Shows "default" ✗
1. Observe that agent continues reporting "default" to manager in heartbeats
1. Restart the agent → Both now show "new-group" (proving TOML was updated correctly)

## Expected Behavior

Both `AgentRPCServer.local_config` and `AbstractAgent.local_config` should reference the same `AgentUnifiedConfig` object. When `update_scaling_group()` updates the scaling group, both should immediately reflect the new value without requiring an agent restart.

## Actual Behavior

Only `AgentRPCServer.local_config` is updated with the new scaling group. `AbstractAgent.local_config` retains the old value because the shared object reference is broken during the update process. The agent continues to report the old scaling group to the manager in heartbeat messages.

## Root Cause

**Location**: `src/ai/backend/agent/server.py:613` (tag version 25.15.5)

**Problematic code**:

```python
def update_scaling_group(self, request: web.Request) -> web.Response:
scaling_group = await request.json()
new_agent_config = self.local_config.agent.model_copy(
update={"scaling_group": scaling_group}
)
# BUG: This creates a NEW AgentUnifiedConfig object
self.local_config = self.local_config.model_copy(update={"agent": new_agent_config})
```

**Technical explanation:**

1. Initial state: Both AgentRPCServer and AbstractAgent reference the same AgentUnifiedConfig object (Object A)

- In single-agent mode: get_agent_configs() returns [self]
- Both share the same config: server.local_config is agent.local_config → True

1. After update: model_copy() creates a new object (Object B)

- AgentRPCServer.local_config → Object B (new scaling group)
- AbstractAgent.local_config → Object A (old scaling group)
- Shared reference broken: server.local_config is agent.local_config → False

1. Consequence: Agent heartbeat reads self.local_config.agent.scaling_group which still points to Object A with the old value

## Solution

```python
def update_scaling_group(self, request: web.Request) -> web.Response:
scaling_group = await request.json()
new_agent_config = self.local_config.agent.model_copy(
update={"scaling_group": scaling_group}
)
# FIX: Only replace the nested agent field, not the entire config object
self.local_config.agent = new_agent_config
```

## Testing

The test verifies:

- Initial state: server.local_config and agent.local_config share the same reference
- After update: Config object reference is preserved (id(config) remains unchanged)
- After update: Both server and agent see the new scaling group value
- TOML file is updated correctly for restart scenarios

Test design:

- Server-level integration test with 9 separated fixtures
- Mocks: AsyncEtcd, AgentStatsPluginContext, AgentErrorPluginContext, RPC components
- Uses shortened temp path to avoid AF_UNIX socket path length issues
- All setup in fixtures; test method contains only test logic

JIRA Issue: BA-3560

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.