kvcache-ai / kvcache-ai/Mooncake

[RFC]: [NoF] Handling stalled SPDK_NVMF I/O

Open
#3,864 2 comments 0 reactions 0 assignees View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Changes proposed

## Background

Mooncake's SPDK_NVMF data path currently has no overall timeout for application I/O. NoF futures for these operations wait indefinitely for completion.

When backend I/O produces neither a completion nor an explicit error, Mooncake read and write calls can remain blocked indefinitely, even if the connection is still healthy.

This can tie up caller threads and I/O resources, causing subsequent requests to queue.

## Expected behavior

When I/O remains unresponsive, Mooncake should return an explicit timeout or failure within a bounded period, while safely handling in-flight I/O and resource cleanup.

## Discussion

1. Should we add a configurable timeout for SPDK_NVMF application I/O?
2. How should Mooncake handle failure reporting, late completions, and resource cleanup for I/O that remains in flight after a timeout?

This RFC outlines the failure scenario and desired behavior. Timeout values, API design, and recovery mechanisms are open for discussion.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Research direction

The issue names no source files, tests, or entry points. Start by locating Mooncake's SPDK_NVMF application I/O path and tracing how NoF futures receive completion or error results. Done requires an agreed configurable timeout, bounded failure reporting, and safe handling of late completions and resource cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.