vllm-project / vllm-project/aibrix
Support session tracking for LLM request
Open
area/gateway
area/inference-engine
help wanted
kind/feature
priority/important-soon
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
RAG and Agent patterns are all multi-thread programs, those application information should be exposed to underneath system to leverage for better colocation etc.
### Use Case
_No response_
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.