kvcache-ai / kvcache-ai/Mooncake
[Bug]: NoF keeps reusing failed qpairs and cannot automatically recover I/O after target recovery
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
## Description
When a connection interruption causes a NoF qpair to fail, Mooncake only marks the request as failed or logs a completion polling error. The cached connection handle remains unchanged.
Subsequent requests to the same endpoint and namespace continue using the failed qpair, causing I/O failures to persist after the target recovers.
## Trigger scenario
1. Start normal NoF reads and writes.
2. Stop the target or interrupt the network until the qpair enters a failed state.
3. Restore connectivity and the target at the same address with the same namespace, then continue I/O from the original client process.
## Expected behavior
NoF reads and writes can recover within the same client process after the target and network recover.
## Current behavior
Based on source code analysis, subsequent requests continue using the cached, failed qpair. Restoring the target alone does not trigger reconnection; recovery typically requires restarting the process holding the failed connection.
## Code references
1. **Cache hits return the existing handle without checking qpair health.**
`SpdkWrapper::OpenNofSegment()` reuses the cached controller. If `ns_seg` already contains the namespace, it immediately returns the existing handle without checking whether its qpair has failed. `spdk_nvme_ctrlr_alloc_io_qpair()` is called only on a namespace cache miss.
[Relevant code](https://github.com/kvcache-ai/Mooncake/blob/e58ad95342158514e273ad287e3859271f079572/mooncake-store/src/spdk/spdk_wrapper.cpp#L312-L359)
2. **Submission failures only update the current task.**
When `SubmitRequest()` returns a nonzero value, `SpdkNofWorkerPool::workerThread()` sets `task->failed = true` and `task->remaining_lba = 0`. It does not invalidate the cached handle or replace the failed qpair.
[Relevant code](https://github.com/kvcache-ai/Mooncake/blob/e58ad95342158514e273ad287e3859271f079572/mooncake-store/src/transfer_task.cpp#L516-L525)
3. **Completion polling failures are only logged.**
When `NvmePollProcessCompletion()` returns a negative value, the worker logs `poll completion error` without recovering the connection or invalidating its cached handle. Subsequent requests therefore continue using the same handle.
[Relevant code](https://github.com/kvcache-ai/Mooncake/blob/e58ad95342158514e273ad287e3859271f079572/mooncake-store/src/transfer_task.cpp#L543-L551)
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Start with SpdkWrapper::OpenNofSegment() in mooncake-store/src/spdk/spdk_wrapper.cpp and the failure paths in SubmitRequest() and NvmePollProcessCompletion() in mooncake-store/src/transfer_task.cpp. Trace how cached namespace handles and qpairs are owned and invalidated, then run the existing NoF I/O tests or reproduce the stop-and-recover scenario. Done means reads and writes recover in the same client process after the target returns, without reusing a failed qpair.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100