kvcache-ai / kvcache-ai/Mooncake
[RFC]: [NoF] Handling stalled SPDK_NVMF I/O
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Changes proposed
## Background
Mooncake's SPDK_NVMF data path currently has no overall timeout for application I/O. NoF futures for these operations wait indefinitely for completion.
When backend I/O produces neither a completion nor an explicit error, Mooncake read and write calls can remain blocked indefinitely, even if the connection is still healthy.
This can tie up caller threads and I/O resources, causing subsequent requests to queue.
## Expected behavior
When I/O remains unresponsive, Mooncake should return an explicit timeout or failure within a bounded period, while safely handling in-flight I/O and resource cleanup.
## Discussion
1. Should we add a configurable timeout for SPDK_NVMF application I/O?
2. How should Mooncake handle failure reporting, late completions, and resource cleanup for I/O that remains in flight after a timeout?
This RFC outlines the failure scenario and desired behavior. Timeout values, API design, and recovery mechanisms are open for discussion.
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Research direction
The issue names no source files, tests, or entry points. Start by locating Mooncake's SPDK_NVMF application I/O path and tracing how NoF futures receive completion or error results. Done requires an agreed configurable timeout, bounded failure reporting, and safe handling of late completions and resource cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100