kvcache-ai / kvcache-ai/Mooncake

[Bug]: Master retains NoF heartbeat probe resources after segment unmount

Open
#4,038 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

## Description

When NoF is enabled, the Master uses its in-process `SpdkWrapper` to connect to remote NVMe-oF namespaces for heartbeat probes. Although the Master does not forward object data, it holds controllers, I/O qpairs, and DMA buffers for probing.

A `NoFSegment` represents a storage range within a specific namespace. Multiple namespaces accessed through the same subsystem and transport address can share a controller, while each namespace has its own probe qpair in the current implementation.

When heartbeat failures trigger `TryUnmountNoFSegmentByHeartbeat()`, the unmount flow removes the segment registration, allocator, related replica metadata, capacity accounting, and heartbeat state. However, it does not release the corresponding resources in `SpdkWrapper`:

- Namespace handle entries and their I/O qpairs in `ns_seg`.
- Probe DMA buffers in `probe_buffers_`.
- Controllers in `connected_ctrlrs` that are no longer used by any namespace.

These resources remain cached until the global `Cleanup()` runs. Explicit unmounts through `UnmountNoFSegment()` have the same issue.

## Impact

For a long-running Master that repeatedly registers and unmounts different namespaces or endpoints, resources associated with segments that have left the storage pool can accumulate, consuming memory, DMA buffers, and queue capacity. If the connections remain active, they may also continue consuming connection and queue resources on the target, potentially affecting subsequent namespace connections.

The same endpoint normally reuses cached resources, so this is not a new connection leak on every registration/unmount cycle.

Additionally, when the same endpoint is registered again, `OpenNofSegment()` returns the cached handle directly. If its qpair has failed, subsequent heartbeat probes may continue failing even after the remote service has recovered.

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Trace TryUnmountNoFSegmentByHeartbeat(), UnmountNoFSegment(), OpenNofSegment(), and SpdkWrapper::Cleanup(), focusing on ns_seg, probe_buffers_, and connected_ctrlrs. Reproduce repeated namespace registration and unmount, then verify that unused namespace handles, qpairs, DMA buffers, and controllers are released. Also verify that re-registering an endpoint does not reuse a failed qpair indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.