lablup / lablup/backend.ai

Add Support for Multiple Agents Within Single 'Agent Runtime' Server

Open
#6,143 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Overview

Enable Backend.AI agent server to support multiple agents within a single physical agent process, now called ‘Agent Runtime’, allowing different scheduling units and backend configurations while maintaining backward compatibility with existing single-agent deployments.

## Background

Currently, Backend.AI uses a 1:1 mapping between physical agent servers and scheduling units. This limits flexibility when:

- Different workload types need different scheduling policies within the same physical machine
- Multiple isolated backend configurations are required on the same host
- Fine-grained resource partitioning is needed beyond the agent level

## Goals

1. **Agent Side**: Support multiple agents with backward compatibility
1. **Manager Side**: Handle mapping agent instances to the correct agent runtime in scheduling and event processing
1. **Gradual Migration**: Enable progressive rollout without disrupting existing deployments

## Architecture Changes

### Agent Server

- Store multiple agents within the same AgentRPCServer
- Add agent ID in server’s RPC functions as now the agent runtime server handles requests for multiple agents
- Ensure legacy single agent configuration works with zero config changes

### Manager Server

- Update scheduler and event handlers to make it handle agent runtime to agent mapping

## Success Criteria

- Existing single-agent deployments continue to work without changes
- New multiple agent deployments can be configured and scheduled correctly
- All agent events (heartbeat, started, terminated) properly interact with agent runtime with many agents
- Scheduler can route requests to correct agents within agent runtime

JIRA Issue: BA-2605

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.