vllm-project / vllm-project/aibrix
[RFC] Support trtllm engine in AIBrix
Open
kind/enhancement
priority/critical-urgent
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
1. router needs to support distributed P/D
2. Env adaptation - cuda13 and GPU driver 580
3. KV Offloading testing and naive support
4. trt-llm metrics support, currently, seem it only supports json style metrics instead of prometheus metrics
5. kv events needs to be supported as well
6. router needs to consume the kv events later
### Use Case
support trt-llm users
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.