vllm-project / vllm-project/aibrix

[Feature] Model Scaling and Request Routing for Multi-LoRA Serving

Open
#2,579 0 comments 0 reactions 1 assignee Claimed by @Suyi32 View on GitHub
area/lora
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

I would like to propose support for model scaling and request routing for Multi-LoRA serving in AIBrix.

This feature would enable AIBrix to scale LoRA adapters at runtime and route requests more effectively, improving flexibility and making LoRA serving more practical and efficient.

### Use Case

This feature would make AIBrix more capable in Multi-LoRA serving scenarios where many LoRA adapters need to be served efficiently on shared infrastructure.

### Proposed Solution

## Task Breakdown

1. Add a Multi-LoRA serving example to the documentation and profile Multi-LoRA serving performance in AIBrix.

- Measure performance tradeoffs, especially TTFT and TPOT latency degradation in Multi-LoRA serving scenarios.

2. Design and implement adapter placement/scaling and LoRA-aware request routing.

- Currently, `ModelAdapter.spec.replicas` behaves as follows:
- If `replicas = 1`, AIBrix skips scoring/routing and routes directly.
- If `replicas` is omitted, the AIBrix gateway load-balances requests among pods.

- To better support Multi-LoRA serving, I propose introducing two mechanisms:
- **Adapter placement policies**: control where adapters are loaded based on base-model identity, `max_lora_rank`, `max_loras`, `max_cpu_loras`, and related constraints. Autoscale LoRA adapters when necessary.
- **LoRA-aware request routing**: make routing decisions based on adapter state and pod load. Routing should prefer pods based on whether the requested adapter is already loaded on GPU and ready, the number of active requests, available capacity, memory pressure, and other relevant signals.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.