vllm-project / vllm-project/aibrix

【Request】KV Management On Huawei NPU

Open
#1,753 0 comments 3 reactions 0 assignees View on GitHub
area/heterogeneous area/kv-cache priority/important-longterm
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

In the field of large language models, the KV offloading feature has become an industry standard, particularly for multi-turn long-dialogue scenarios, as it significantly reduces TTFT.
With the increasing adoption of domestic chips in the future, our focus here is on how to adapt this functionality to NPUs on AIbrix.

### Use Case

Based on AIbrix's current plug-and-play KV cache management framework, integrate NPU hardware with the following capabilities:
1. Compatible with SGLang/vLLM
2. Support single-node/distributed offloading
3. Support multiple storage backends
4. Support multiple KV management strategies
5. Decouple KV offload from placement decisions (implemented via centralized storage)

### Proposed Solution

I would suggest implementing this through a modular layered approach:
1.Inference Engine Layer: Adapt the AIbrix KV Connector for NPUs at this level.
2.AIbrix Layer: Integrate NPUs and related operators into the KV Management framework within this layer.
3.Hardware Layer:
3.1 Implement NPU KV read/write operators.
3.2 Handle network-related adaptations, potentially requiring a protocol conversion layer (as NPUs may use proprietary protocols, necessitating an intermediate layer to interface with RDMA or deployment in fully NPU-based machines).
4.Storage Layer: Ensure compatibility and adaptation for NPU-based machines.

Contributor guide

Open the contributing guide

Research direction

The payload names the AIbrix KV Connector and KV Management framework as entry points; begin by tracing how the existing plug-and-play KV cache management framework connects to SGLang/vLLM. Done means an NPU adaptation covering single-node and distributed offloading, storage backends, management strategies, and the required network and storage integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai-infra-agents, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.