vllm-project / vllm-project/aibrix
Does aibrix support to do load balance against managed model endpoints
Open
area/gateway
triage/needs-information
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
For example, is it possible to use aibrix gateway to do load balance for azure openai endpoints so that we can take advantage of the gateway features like prefix cache load balance.
Contributor guide
Assessment
This issue has not been assessed yet.