kvcache-ai / kvcache-ai/Mooncake
[Bug]: NoF master keeps allocating space despite client-to-target network partitions
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
## Description
NoF heartbeats are initiated by the master. If the master can reach a NoF target while a client cannot, the master may continue allocating space on that target even though the client cannot complete the actual I/O.
## Scenario
```text
Client ←── Metadata RPC: healthy ──→ Master
│
│ NoF heartbeat: healthy
↓
Client ←──── Network partition ────→ NoF Target
```
In this scenario, the client can successfully issue `PutStart`, and the master's NoF probes continue to succeed, leaving the segment available for allocation. However, after receiving the allocation, the client cannot access the target. The write consequently fails or waits for the underlying transport to report an error.
Because the master's probes continue to succeed, this condition may persist throughout the partition rather than being limited to the heartbeat detection window.
## Current code behavior
Based on the `USE_NOF` / SPDK path at commit `cafc5078`:
- **Heartbeats verify the master-to-target path.** `ProbeNofSegment()` reads LBA 0 of the target namespace through the master's own SPDK connection. This does not establish reachability from a client. See the [probe implementation](https://github.com/kvcache-ai/Mooncake/blob/cafc5078/mooncake-store/src/spdk/spdk_wrapper.cpp#L423).
- **NoF allocation does not filter targets by client reachability.** The master allocates space from mounted NoF allocators without excluding targets that are unreachable from the requesting client. See the [allocation logic](https://github.com/kvcache-ai/Mooncake/blob/cafc5078/mooncake-store/src/master_service.cpp#L4123).
- **A client-side I/O failure does not update reachability information for subsequent allocations.** The write path calls `PutEnd` or `PutRevoke` according to transfer results. `PutRevoke` handles replicas of the current object; it does not mark the target as unreachable from that client. Subsequent requests may therefore receive allocations on the same target. See the [client write path](https://github.com/kvcache-ai/Mooncake/blob/cafc5078/mooncake-store/src/client_service.cpp#L1881) and [PutRevoke implementation](https://github.com/kvcache-ai/Mooncake/blob/cafc5078/mooncake-store/src/master_service.cpp#L4776).
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Start with ProbeNofSegment() in mooncake-store/src/spdk/spdk_wrapper.cpp, then trace NoF allocation in mooncake-store/src/master_service.cpp and the client write path in mooncake-store/src/client_service.cpp. Review PutRevoke in master_service.cpp and the referenced commit to understand existing reachability state. Done means a client that cannot reach a target is not repeatedly allocated space there, while healthy client-to-target paths continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100