kvcache-ai / kvcache-ai/Mooncake

[Bug]: client 重启后, ssd 上的数据全部失效

Open
#3,052 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report
mooncake version: v0.3.11.post1

**1. Background**
When a client with SSD offload enabled crashes unexpectedly and is subsequently restarted in a short timeframe, the SSD data previously held by the client becomes stale/invalid, rendering it inaccessible.
**2. Analysis**
Under normal circumstances, upon restart, the client is expected to scan the data residing on the SSD and report the metadata to the Master via ScanMeta. However, if the new client's registration/reporting occurs within the TTL (Time-To-Live) window of the previous client instance — meaning the old client has not yet been evicted by the Master — the new client's registration attempt is effectively a no-op and is rejected. The root cause of this issue resides in the logic of auto MasterService::AddReplica(const UUID& client_id, const std::string& key, Replica& replica).
```c++
auto MasterService::AddReplica(const UUID& client_id, const std::string& key,
Replica& replica)
-> tl::expected {
std::shared_lock shared_lock(snapshot_mutex_);
MetadataAccessorRW accessor(this, key);
if (!accessor.Exists()) {
accessor.Create(
client_id,
replica.get_descriptor().get_local_disk_descriptor().object_size,
std::vector{}, false);
}
auto& metadata = accessor.Get();
if (replica.type() != ReplicaType::LOCAL_DISK) {
LOG(ERROR) << "Invalid replica type: " << replica.type()
<< ". Expected ReplicaType::LOCAL_DISK.";
return tl::make_unexpected(ErrorCode::INVALID_PARAMS);
}

if (!metadata.HasReplica(&Replica::fn_is_local_disk_replica)) {
std::vector replicas;
replicas.emplace_back(std::move(replica));
metadata.AddReplicas(std::move(replicas));
return {};
}

metadata.VisitReplicas(
[client_id](const Replica& rep) {
return rep.type() == ReplicaType::LOCAL_DISK &&
rep.get_descriptor().get_local_disk_descriptor().client_id ==
client_id;
},
[&replica](Replica& rep) {
rep.get_descriptor()
.get_local_disk_descriptor()
.transport_endpoint = replica.get_descriptor()
.get_local_disk_descriptor()
.transport_endpoint;
rep.get_descriptor().get_local_disk_descriptor().object_size =
replica.get_descriptor()
.get_local_disk_descriptor()
.object_size;
});
return {};
}
```
When the new client reports its keys, since the old client has not yet expired, those keys already exist in the Master's metadata. During the check for whether a disk replica exists for the keys reported by the new client, there is no validation to verify whether the client_id matches the one associated with the existing metadata entry. As a result, the Master incorrectly concludes that the data reported by the new client is already present and takes no further action.

Subsequently, when the old client reaches its TTL and is evicted by the Master, its corresponding metadata is also purged. Consequently, the KV cache data residing on the SSD becomes orphaned and can no longer be accessed or served.

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Start at MasterService::AddReplica(const UUID& client_id, const std::string& key, Replica& replica) and trace the metadata HasReplica and VisitReplicas checks, along with the client restart and ScanMeta flow. Done means a restarted client’s SSD keys are associated with its client_id and remain accessible after the old client’s TTL eviction, with regression coverage for this sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.