kvcache-ai / kvcache-ai/Mooncake
[Bug]: client 重启后, ssd 上的数据全部失效
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
mooncake version: v0.3.11.post1
**1. Background**
When a client with SSD offload enabled crashes unexpectedly and is subsequently restarted in a short timeframe, the SSD data previously held by the client becomes stale/invalid, rendering it inaccessible.
**2. Analysis**
Under normal circumstances, upon restart, the client is expected to scan the data residing on the SSD and report the metadata to the Master via ScanMeta. However, if the new client's registration/reporting occurs within the TTL (Time-To-Live) window of the previous client instance — meaning the old client has not yet been evicted by the Master — the new client's registration attempt is effectively a no-op and is rejected. The root cause of this issue resides in the logic of auto MasterService::AddReplica(const UUID& client_id, const std::string& key, Replica& replica).
```c++
auto MasterService::AddReplica(const UUID& client_id, const std::string& key,
Replica& replica)
-> tl::expected {
std::shared_lock shared_lock(snapshot_mutex_);
MetadataAccessorRW accessor(this, key);
if (!accessor.Exists()) {
accessor.Create(
client_id,
replica.get_descriptor().get_local_disk_descriptor().object_size,
std::vector{}, false);
}
auto& metadata = accessor.Get();
if (replica.type() != ReplicaType::LOCAL_DISK) {
LOG(ERROR) << "Invalid replica type: " << replica.type()
<< ". Expected ReplicaType::LOCAL_DISK.";
return tl::make_unexpected(ErrorCode::INVALID_PARAMS);
}
if (!metadata.HasReplica(&Replica::fn_is_local_disk_replica)) {
std::vector replicas;
replicas.emplace_back(std::move(replica));
metadata.AddReplicas(std::move(replicas));
return {};
}
metadata.VisitReplicas(
[client_id](const Replica& rep) {
return rep.type() == ReplicaType::LOCAL_DISK &&
rep.get_descriptor().get_local_disk_descriptor().client_id ==
client_id;
},
[&replica](Replica& rep) {
rep.get_descriptor()
.get_local_disk_descriptor()
.transport_endpoint = replica.get_descriptor()
.get_local_disk_descriptor()
.transport_endpoint;
rep.get_descriptor().get_local_disk_descriptor().object_size =
replica.get_descriptor()
.get_local_disk_descriptor()
.object_size;
});
return {};
}
```
When the new client reports its keys, since the old client has not yet expired, those keys already exist in the Master's metadata. During the check for whether a disk replica exists for the keys reported by the new client, there is no validation to verify whether the client_id matches the one associated with the existing metadata entry. As a result, the Master incorrectly concludes that the data reported by the new client is already present and takes no further action.
Subsequently, when the old client reaches its TTL and is evicted by the Master, its corresponding metadata is also purged. Consequently, the KV cache data residing on the SSD becomes orphaned and can no longer be accessed or served.
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Start at MasterService::AddReplica(const UUID& client_id, const std::string& key, Replica& replica) and trace the metadata HasReplica and VisitReplicas checks, along with the client restart and ScanMeta flow. Done means a restarted client’s SSD keys are associated with its client_id and remain accessible after the old client’s TTL eviction, with regression coverage for this sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100