kvcache-ai / kvcache-ai/Mooncake

[Bug]: Snapshot restore loses segment protocol and allocator replica_type

Open
#2,636 3 comments 1 reaction 2 assignees Claimed by @Icedcoco View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

### Description

After master restart with snapshot restore (`enable_snapshot_restore=1`), all memory segments lose their `protocol` field. The admin API returns `"protocol": ""` even though the segment was originally
mounted with `"protocol": "tcp"`.

Additionally, `OffsetBufferAllocator::replica_type_` is also lost during serialization, which affects NoF segment metric accounting (restored NoF segments are counted under "mem" instead of "nof").

### Reproduction

1. Start Mooncake master with snapshot enabled
2. Mount a segment (e.g. protocol=tcp)
3. Wait for a snapshot to be generated
4. Restart the master
5. Query the admin API:

```bash
curl http://${MOONCAKE_MASTER_URL}/get_segments_detail | python3 -m json.tool

Output:

{
"total_segments": 1,
"segments": [
{
"segment_name": "xxx.xxx.xxx.148:13695",
"segment_id": "6649975730718913833-14683276544698645121",
"client_id": "9101436600682018864-15666146750897147799",
"base_address": "0x773c56000000",
"size_bytes": 3355443200,
"size_human": "3.125 GiB",
"te_endpoint": "xxx.xxx.xxx.148:13695",
"protocol": "", # ← empty after restore
"status": "OK",
"allocator_used_bytes": 3165446144,
"allocator_capacity_bytes": 3355443200,
"allocator_usage_percent": 94.34
}
]
}
```
Note that protocol is empty even though the segment was mounted with --protocol tcp.

### Root Cause

The snapshot serialization code in serializer.cpp uses custom msgpack serializers (not YLT_REFL auto-serialization) for both MountedSegment and OffsetBufferAllocator. These custom serializers missed two
fields:

1. MountedSegment::segment.protocol missing

Serialize (serializer.cpp:829-868) packs 8 elements:
- segment.id, segment.name, segment.base, segment.size, segment.te_endpoint
- status, has_buffer_allocator, buffer_allocator_data

But the Segment struct has 6 fields: id, name, base, size, te_endpoint, protocol. protocol is never serialized.

After restore, segment.protocol defaults to empty string "".

2. OffsetBufferAllocator::replica_type_ missing

Serialize (serializer.cpp:930-954) packs 6 elements:
- segment_name, base, total_size, cur_size, transport_endpoint, offset_allocator

But OffsetBufferAllocator has replica_type_ which determines whether metrics are counted as MEMORY or NOF_SSD. replica_type_ is never serialized.

After restore, the allocator is reconstructed with the default ReplicaType::MEMORY, so any NoF segment's metrics are incorrectly counted under memory.

### Fix Options

Option A: Add missing fields to serialization (no backward compat)

Add the fields at the natural position in the serialization arrays:

- MountedSegment: add protocol after te_endpoint in both serialize/deserialize, change array size from 8→9
- OffsetBufferAllocator: add replica_type_ after transport_endpoint_ in both serialize/deserialize, change array size from 6→7

Old snapshots become unreadable, but given that the restore cleanup bug (#TODO) already makes old snapshots unreliable, this is acceptable.

Option B: Backward-compatible fix (more complex)

Append the new fields at the end of the arrays and add size detection in deserialization:
- Deserialize accepts both old (8) and new (9) element counts for MountedSegment
- Deserialize accepts both old (6) and new (7) element counts for OffsetBufferAllocator

This preserves the ability to read old snapshots but adds complexity.

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.