kvcache-ai / kvcache-ai/Mooncake
[RFC]: Add NoF initiator multipath support through SPDK NVMe bdev
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
## Changes proposed
The Mooncake Store NoF initiator currently uses SPDK NVMe APIs directly to connect to targets and submit I/O. It does not support multiple paths to the same namespace. If a network link, NIC, or target endpoint fails, the initiator cannot automatically switch to an alternative path, even when the namespace remains accessible through that path.
It would be useful to support path redundancy: connect to the same namespace through multiple endpoints, automatically fail over to another available path when one fails, and reconnect recovered paths. The primary focus of this discussion is fault tolerance; load balancing could be considered separately.
I propose evaluating the [SPDK NVMe bdev layer](https://spdk.io/doc/nvme_multipath.html) as the foundation for multipath support in the NoF initiator. This layer already provides multipath, ANA handling, controller reset, reconnection, and I/O retry mechanisms. It also manages underlying qpairs through per-thread I/O channels. Reusing these capabilities could reduce the queue lifecycle, recovery, and path management logic maintained by Mooncake.
Feedback and suggestions are welcome, especially on the feasibility of this approach, potential trade-offs, and other ways to provide multipath fault tolerance for the NoF initiator.
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Research direction
Start by reading the SPDK NVMe multipath documentation linked in the issue, then inspect the NoF initiator’s current direct SPDK NVMe API usage. Done means producing an agreed approach for connecting multiple endpoints, failing over, reconnecting recovered paths, and defining the scope separately from load balancing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- infrastructure, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100