Support LoRA adapters for vLLM worker
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
vLLM has recently introduced experimental support for LoRA adapters both via code and API server: https://docs.vllm.ai/en/latest/models/lora.html. It would be interesting to add LoRA adapter support to the vLLM worker as well.
Ideally, the worker should be able to serve requests from the base model and 0-to-many LoRAs. However, serving with just a single LoRA would be good enough initially.
I might be able to contribute, though I am not very familiar with FastChat code.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the vLLM LoRA documentation linked in the issue and locating the vLLM worker entry point in FastChat. Trace how the worker currently serves the base model and how requests reach it. Done means the worker can serve the base model with at least one LoRA adapter, with support for multiple adapters left as a possible extension.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100