kvcache-ai / kvcache-ai/Mooncake

[Feature Request]: Support GPU buffers in Mooncake Store NOF reads and writes

Open
#4,128 2 comments 0 reactions 2 assignees Claimed by @fcczzz View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Describe your feature request

Clients such as vLLM keep KV cache data in GPU memory and pass GPU buffers to Mooncake Store Client. The existing NOF path uses SPDK for I/O and requires contiguous CPU DMA buffers, so it cannot directly handle these GPU pointers.

Add CPU staging in Store Client to support the following data paths:

- PUT: Gather GPU slices into a contiguous CPU DMA buffer, then write it to NOF.
- GET: Read NOF data into a CPU DMA buffer, then scatter it into the destination GPU slices.

The client should select staging automatically when a NOF replica is used with GPU buffers, and support non-contiguous GPU slices and batched reads and writes.

This would allow clients such as vLLM to use the NOF backend through the native Mooncake Store PUT/GET APIs.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.