vllm-project / vllm-project/aibrix
Track production-grade Serverless LoRA support
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 697
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
### 🚀 Feature Description and Motivation
We already support multi-LoRA / high-density LoRA deployment, but our current design is closer to “serverful” LLM serving. For true serverless LoRA scenarios (bursty, fine-grained, pay-as-you-go), we still lack:
- Efficient sharing of the same backbone model across many LoRA “functions”
- Systematic handling of LoRA artifact cold starts (adapters, libs, JIT kernels)
- Contention-aware resource management when many LoRAs compete on the same GPU
The paper ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs https://arxiv.org/abs/2505.14468 proposes a serverless inference system that addresses these by:
- secure backbone LLM sharing,
- comprehensive LoRA artifact pre-loading, and
- contention-aware batching + dynamic GPU offloading.
We want to borrow these ideas and make “Serverless LoRA” a first-class production capability in AIBrix.
Beside the paper idea, we also have some other known issues to resolve
- [x] https://github.com/vllm-project/aibrix/issues/1825
- [x] https://github.com/vllm-project/aibrix/issues/1819
- [ ] https://github.com/vllm-project/aibrix/issues/1826
- Engine behavior testing.
- [ ] https://github.com/vllm-project/aibrix/issues/1831
### Use Case
production grace lora feature
### Proposed Solution
_No response_
Contributor guide
Research direction
Start with the ServerlessLoRA paper and the unresolved dependency issues #1826 and #1831; the issue names no source files or entry points. Review the existing multi-LoRA/high-density deployment and engine behavior testing to establish the current baseline. Done should mean a concrete, agreed production-scope plan with tests and acceptance criteria for serverless LoRA behavior.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, cloud, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100