vllm-project / vllm-project/aibrix
[router] LSH based prefix cache aware router
Open
area/gateway
priority/important-soon
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
Right now, we're using xxhash in https://github.com/aibrix/aibrix/pull/641 for our prefix cache-aware router. We might consider switching to a consistent hash + LSH-based approach, which could reduce accuracy a bit but would simplify scaling. Here are some related discussions: https://github.com/vllm-project/production-stack/issues/59#issuecomment-2656740442.
### Use Case
N/A
### Proposed Solution
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.