NVIDIA / NVIDIA/TensorRT-Edge-LLM
支持多lora同时运行
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
背景
目前Tensorrt-Edge-LLM对lora的支持只能达到同一个lora的多个请求合并到一个batch的程度，并且只能跑一个lora。
期望
1、可以支持同时加载多个Lora到显存，并支持同时运行多个Lora
2、同一个batch里不同请求可以指定不同的Lora，显存里被指定的Lora才运行
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by locating the existing LoRA loading and batching implementation, then trace how requests select a LoRA. Done means multiple LoRAs can remain loaded and run concurrently, with different requests in one batch selecting different loaded LoRAs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100