kvcache-ai / kvcache-ai/Mooncake
[Bug]: Snapshot restore loses segment protocol and allocator replica_type
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
### Description
After master restart with snapshot restore (`enable_snapshot_restore=1`), all memory segments lose their `protocol` field. The admin API returns `"protocol": ""` even though the segment was originally
mounted with `"protocol": "tcp"`.
Additionally, `OffsetBufferAllocator::replica_type_` is also lost during serialization, which affects NoF segment metric accounting (restored NoF segments are counted under "mem" instead of "nof").
### Reproduction
1. Start Mooncake master with snapshot enabled
2. Mount a segment (e.g. protocol=tcp)
3. Wait for a snapshot to be generated
4. Restart the master
5. Query the admin API:
```bash
curl http://${MOONCAKE_MASTER_URL}/get_segments_detail | python3 -m json.tool
Output:
{
"total_segments": 1,
"segments": [
{
"segment_name": "xxx.xxx.xxx.148:13695",
"segment_id": "6649975730718913833-14683276544698645121",
"client_id": "9101436600682018864-15666146750897147799",
"base_address": "0x773c56000000",
"size_bytes": 3355443200,
"size_human": "3.125 GiB",
"te_endpoint": "xxx.xxx.xxx.148:13695",
"protocol": "", # ← empty after restore
"status": "OK",
"allocator_used_bytes": 3165446144,
"allocator_capacity_bytes": 3355443200,
"allocator_usage_percent": 94.34
}
]
}
```
Note that protocol is empty even though the segment was mounted with --protocol tcp.
### Root Cause
The snapshot serialization code in serializer.cpp uses custom msgpack serializers (not YLT_REFL auto-serialization) for both MountedSegment and OffsetBufferAllocator. These custom serializers missed two
fields:
1. MountedSegment::segment.protocol missing
Serialize (serializer.cpp:829-868) packs 8 elements:
- segment.id, segment.name, segment.base, segment.size, segment.te_endpoint
- status, has_buffer_allocator, buffer_allocator_data
But the Segment struct has 6 fields: id, name, base, size, te_endpoint, protocol. protocol is never serialized.
After restore, segment.protocol defaults to empty string "".
2. OffsetBufferAllocator::replica_type_ missing
Serialize (serializer.cpp:930-954) packs 6 elements:
- segment_name, base, total_size, cur_size, transport_endpoint, offset_allocator
But OffsetBufferAllocator has replica_type_ which determines whether metrics are counted as MEMORY or NOF_SSD. replica_type_ is never serialized.
After restore, the allocator is reconstructed with the default ReplicaType::MEMORY, so any NoF segment's metrics are incorrectly counted under memory.
### Fix Options
Option A: Add missing fields to serialization (no backward compat)
Add the fields at the natural position in the serialization arrays:
- MountedSegment: add protocol after te_endpoint in both serialize/deserialize, change array size from 8→9
- OffsetBufferAllocator: add replica_type_ after transport_endpoint_ in both serialize/deserialize, change array size from 6→7
Old snapshots become unreadable, but given that the restore cleanup bug (#TODO) already makes old snapshots unreliable, this is acceptable.
Option B: Backward-compatible fix (more complex)
Append the new fields at the end of the arrays and add size detection in deserialization:
- Deserialize accepts both old (8) and new (9) element counts for MountedSegment
- Deserialize accepts both old (6) and new (7) element counts for OffsetBufferAllocator
This preserves the ability to read old snapshots but adds complexity.
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Assessment
This issue has not been assessed yet.