vllm-project / vllm-project/aibrix
Implement cold start manager for lora models
Open
area/lora
kind/feature
priority/important-longterm
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
cache locality can be leveraged to reduce model startup time. As user uses up to 128 rank which is kind of large, this feature would be helpful. this is from https://github.com/aibrix/aibrix/issues/22
### Use Case
_No response_
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.