vllm-project / vllm-project/aibrix
【Request】KV Management On Huawei NPU
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🚀 Feature Description and Motivation
In the field of large language models, the KV offloading feature has become an industry standard, particularly for multi-turn long-dialogue scenarios, as it significantly reduces TTFT.
With the increasing adoption of domestic chips in the future, our focus here is on how to adapt this functionality to NPUs on AIbrix.
### Use Case
Based on AIbrix's current plug-and-play KV cache management framework, integrate NPU hardware with the following capabilities:
1. Compatible with SGLang/vLLM
2. Support single-node/distributed offloading
3. Support multiple storage backends
4. Support multiple KV management strategies
5. Decouple KV offload from placement decisions (implemented via centralized storage)
### Proposed Solution
I would suggest implementing this through a modular layered approach:
1.Inference Engine Layer: Adapt the AIbrix KV Connector for NPUs at this level.
2.AIbrix Layer: Integrate NPUs and related operators into the KV Management framework within this layer.
3.Hardware Layer:
3.1 Implement NPU KV read/write operators.
3.2 Handle network-related adaptations, potentially requiring a protocol conversion layer (as NPUs may use proprietary protocols, necessitating an intermediate layer to interface with RDMA or deployment in fully NPU-based machines).
4.Storage Layer: Ensure compatibility and adaptation for NPU-based machines.
Contributor guide
Research direction
The payload names the AIbrix KV Connector and KV Management framework as entry points; begin by tracing how the existing plug-and-play KV cache management framework connects to SGLang/vLLM. Done means an NPU adaptation covering single-node and distributed offloading, storage backends, management strategies, and the required network and storage integration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai-infra-agents, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100