vllm-project / vllm-project/aibrix

[RFC] Support trtllm engine in AIBrix

Open
#2,001 2 comments 0 reactions 3 assignees Claimed by @DwyaneShi View on GitHub
kind/enhancement priority/critical-urgent
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

1. router needs to support distributed P/D
2. Env adaptation - cuda13 and GPU driver 580
3. KV Offloading testing and naive support
4. trt-llm metrics support, currently, seem it only supports json style metrics instead of prometheus metrics
5. kv events needs to be supported as well
6. router needs to consume the kv events later

### Use Case

support trt-llm users

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.