kvcache-ai / kvcache-ai/Mooncake
[Bug]: Master retains NoF heartbeat probe resources after segment unmount
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
## Description
When NoF is enabled, the Master uses its in-process `SpdkWrapper` to connect to remote NVMe-oF namespaces for heartbeat probes. Although the Master does not forward object data, it holds controllers, I/O qpairs, and DMA buffers for probing.
A `NoFSegment` represents a storage range within a specific namespace. Multiple namespaces accessed through the same subsystem and transport address can share a controller, while each namespace has its own probe qpair in the current implementation.
When heartbeat failures trigger `TryUnmountNoFSegmentByHeartbeat()`, the unmount flow removes the segment registration, allocator, related replica metadata, capacity accounting, and heartbeat state. However, it does not release the corresponding resources in `SpdkWrapper`:
- Namespace handle entries and their I/O qpairs in `ns_seg`.
- Probe DMA buffers in `probe_buffers_`.
- Controllers in `connected_ctrlrs` that are no longer used by any namespace.
These resources remain cached until the global `Cleanup()` runs. Explicit unmounts through `UnmountNoFSegment()` have the same issue.
## Impact
For a long-running Master that repeatedly registers and unmounts different namespaces or endpoints, resources associated with segments that have left the storage pool can accumulate, consuming memory, DMA buffers, and queue capacity. If the connections remain active, they may also continue consuming connection and queue resources on the target, potentially affecting subsequent namespace connections.
The same endpoint normally reuses cached resources, so this is not a new connection leak on every registration/unmount cycle.
Additionally, when the same endpoint is registered again, `OpenNofSegment()` returns the cached handle directly. If its qpair has failed, subsequent heartbeat probes may continue failing even after the remote service has recovered.
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Trace TryUnmountNoFSegmentByHeartbeat(), UnmountNoFSegment(), OpenNofSegment(), and SpdkWrapper::Cleanup(), focusing on ns_seg, probe_buffers_, and connected_ctrlrs. Reproduce repeated namespace registration and unmount, then verify that unused namespace handles, qpairs, DMA buffers, and controllers are released. Also verify that re-registering an endpoint does not reuse a failed qpair indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100