kvcache-ai / kvcache-ai/Mooncake

[Bug]: NoF keeps reusing failed qpairs and cannot automatically recover I/O after target recovery

Open
#3,863 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

## Description

When a connection interruption causes a NoF qpair to fail, Mooncake only marks the request as failed or logs a completion polling error. The cached connection handle remains unchanged.

Subsequent requests to the same endpoint and namespace continue using the failed qpair, causing I/O failures to persist after the target recovers.

## Trigger scenario

1. Start normal NoF reads and writes.
2. Stop the target or interrupt the network until the qpair enters a failed state.
3. Restore connectivity and the target at the same address with the same namespace, then continue I/O from the original client process.

## Expected behavior

NoF reads and writes can recover within the same client process after the target and network recover.

## Current behavior

Based on source code analysis, subsequent requests continue using the cached, failed qpair. Restoring the target alone does not trigger reconnection; recovery typically requires restarting the process holding the failed connection.

## Code references

1. **Cache hits return the existing handle without checking qpair health.**

`SpdkWrapper::OpenNofSegment()` reuses the cached controller. If `ns_seg` already contains the namespace, it immediately returns the existing handle without checking whether its qpair has failed. `spdk_nvme_ctrlr_alloc_io_qpair()` is called only on a namespace cache miss.

[Relevant code](https://github.com/kvcache-ai/Mooncake/blob/e58ad95342158514e273ad287e3859271f079572/mooncake-store/src/spdk/spdk_wrapper.cpp#L312-L359)

2. **Submission failures only update the current task.**

When `SubmitRequest()` returns a nonzero value, `SpdkNofWorkerPool::workerThread()` sets `task->failed = true` and `task->remaining_lba = 0`. It does not invalidate the cached handle or replace the failed qpair.

[Relevant code](https://github.com/kvcache-ai/Mooncake/blob/e58ad95342158514e273ad287e3859271f079572/mooncake-store/src/transfer_task.cpp#L516-L525)

3. **Completion polling failures are only logged.**

When `NvmePollProcessCompletion()` returns a negative value, the worker logs `poll completion error` without recovering the connection or invalidating its cached handle. Subsequent requests therefore continue using the same handle.

[Relevant code](https://github.com/kvcache-ai/Mooncake/blob/e58ad95342158514e273ad287e3859271f079572/mooncake-store/src/transfer_task.cpp#L543-L551)

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Start with SpdkWrapper::OpenNofSegment() in mooncake-store/src/spdk/spdk_wrapper.cpp and the failure paths in SubmitRequest() and NvmePollProcessCompletion() in mooncake-store/src/transfer_task.cpp. Trace how cached namespace handles and qpairs are owned and invalidated, then run the existing NoF I/O tests or reproduce the stop-and-recover scenario. Done means reads and writes recover in the same client process after the target returns, without reusing a failed qpair.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.