kvcache-ai / kvcache-ai/Mooncake

[RFC]: Add NoF initiator multipath support through SPDK NVMe bdev

Open
#4,050 1 comment 0 reactions 0 assignees View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Changes proposed

The Mooncake Store NoF initiator currently uses SPDK NVMe APIs directly to connect to targets and submit I/O. It does not support multiple paths to the same namespace. If a network link, NIC, or target endpoint fails, the initiator cannot automatically switch to an alternative path, even when the namespace remains accessible through that path.

It would be useful to support path redundancy: connect to the same namespace through multiple endpoints, automatically fail over to another available path when one fails, and reconnect recovered paths. The primary focus of this discussion is fault tolerance; load balancing could be considered separately.

I propose evaluating the [SPDK NVMe bdev layer](https://spdk.io/doc/nvme_multipath.html) as the foundation for multipath support in the NoF initiator. This layer already provides multipath, ANA handling, controller reset, reconnection, and I/O retry mechanisms. It also manages underlying qpairs through per-thread I/O channels. Reusing these capabilities could reduce the queue lifecycle, recovery, and path management logic maintained by Mooncake.

Feedback and suggestions are welcome, especially on the feasibility of this approach, potential trade-offs, and other ways to provide multipath fault tolerance for the NoF initiator.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Research direction

Start by reading the SPDK NVMe multipath documentation linked in the issue, then inspect the NoF initiator’s current direct SPDK NVMe API usage. Done means producing an agreed approach for connecting multiple endpoints, failing over, reconnecting recovered paths, and defining the scope separately from load balancing.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
infrastructure, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.