vllm-project / vllm-project/aibrix
Intercept the request to model registry and forward to runtime
Open
area/acceleration
kind/feature
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
Current runtime model downloading way won't be accepted by all the users. In order to increase the adoption rate, I am thinking whether there's more intelligent way. e.g. intercept the request and forward to the proxy. In that case, user doesn't need to make any changes and we can inject some proxy address to archives the similar performance
### Use Case
_No response_
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.