vllm-project / vllm-project/aibrix

[router] LSH based prefix cache aware router

Open
#672 6 comments 0 reactions 0 assignees View on GitHub
area/gateway priority/important-soon
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

Right now, we're using xxhash in https://github.com/aibrix/aibrix/pull/641 for our prefix cache-aware router. We might consider switching to a consistent hash + LSH-based approach, which could reduce accuracy a bit but would simplify scaling. Here are some related discussions: https://github.com/vllm-project/production-stack/issues/59#issuecomment-2656740442.

### Use Case

N/A

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.