vllm-project / vllm-project/aibrix

Track production-grade Serverless LoRA support

Open
#1,818 0 comments 2 reactions 0 assignees View on GitHub
area/lora kind/feature priority/critical-urgent
Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🚀 Feature Description and Motivation

We already support multi-LoRA / high-density LoRA deployment, but our current design is closer to “serverful” LLM serving. For true serverless LoRA scenarios (bursty, fine-grained, pay-as-you-go), we still lack:

- Efficient sharing of the same backbone model across many LoRA “functions”
- Systematic handling of LoRA artifact cold starts (adapters, libs, JIT kernels)
- Contention-aware resource management when many LoRAs compete on the same GPU

The paper ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs https://arxiv.org/abs/2505.14468 proposes a serverless inference system that addresses these by:
- secure backbone LLM sharing,
- comprehensive LoRA artifact pre-loading, and
- contention-aware batching + dynamic GPU offloading.

We want to borrow these ideas and make “Serverless LoRA” a first-class production capability in AIBrix.

Beside the paper idea, we also have some other known issues to resolve
- [x] https://github.com/vllm-project/aibrix/issues/1825
- [x] https://github.com/vllm-project/aibrix/issues/1819
- [ ] https://github.com/vllm-project/aibrix/issues/1826
- Engine behavior testing.
- [ ] https://github.com/vllm-project/aibrix/issues/1831

### Use Case

production grace lora feature

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the ServerlessLoRA paper and the unresolved dependency issues #1826 and #1831; the issue names no source files or entry points. Review the existing multi-LoRA/high-density deployment and engine behavior testing to establish the current baseline. Done should mean a concrete, agreed production-scope plan with tests and acceptance criteria for serverless LoRA behavior.

Written by the indexing model from the issue text.

Assessment

Domain
ai, cloud, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.