kvcache-ai / kvcache-ai/Mooncake
[Feature Request]: Support GPU buffers in Mooncake Store NOF reads and writes
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Describe your feature request
Clients such as vLLM keep KV cache data in GPU memory and pass GPU buffers to Mooncake Store Client. The existing NOF path uses SPDK for I/O and requires contiguous CPU DMA buffers, so it cannot directly handle these GPU pointers.
Add CPU staging in Store Client to support the following data paths:
- PUT: Gather GPU slices into a contiguous CPU DMA buffer, then write it to NOF.
- GET: Read NOF data into a CPU DMA buffer, then scatter it into the destination GPU slices.
The client should select staging automatically when a NOF replica is used with GPU buffers, and support non-contiguous GPU slices and batched reads and writes.
This would allow clients such as vLLM to use the NOF backend through the native Mooncake Store PUT/GET APIs.
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Assessment
This issue has not been assessed yet.