kvcache-ai / kvcache-ai/Mooncake

[Bug]: HA模式切换新leader后元数据丢失

Open
#1,871 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

在三个节点上分别部署etcd、mooncake master,拉起vllm服务并发送请求,master pool中有kvcache的内存占用,手动kill master leader切换到新leader后,元数据丢失,新的master pool中kvcache的内存为0。
具体操作步骤如下:

1. 打开STORE_USE_ETCD编译mooncake
```
git clone -b v0.3.9 --depth 1 https://github.com/kvcache-ai/Mooncake.git
cd Mooncake
git submodule update --init --recursive # 解决报错 "Could not find pybind11"
mkdir build
cd build
cmake .. -DUSE_ASCEND_DIRECT=ON -DSTORE_USE_ETCD=ON
make -j
make install
```

2. 在三个节点上分别执行以下命令启动etcd,name和ip修改成各自的:
```
etcd \
--name etcd_node0 \
--data-dir /data/liaobiting/mooncake/etcd \
--listen-client-urls http://0.0.0.0:2379 \
--advertise-client-urls http://100.100.135.188:2379 \
--listen-peer-urls http://0.0.0.0:2380 \
--initial-advertise-peer-urls http://100.100.135.188:2380 \
--initial-cluster etcd_node0=http://100.100.135.188:2380,etcd_node1=http://100.100.135.184:2380,etcd_node2=http://100.100.135.174:2380 \
--initial-cluster-state new \
--initial-cluster-token etcd-cluster-1
```

3. 在三个节点上分别执行以下命令启动master,rpc-address修改成各自的ip:
```
mooncake_master --enable-ha true --etcd-endpoints "100.100.135.188:2379;100.100.135.184:2379;100.100.135.174:2379" --rpc-address 100.100.135.188 --rpc-port 50051
```

4. 在三个节点上配置mooncake_config.json,local_hostname改成各自的ip:
```
{
"local_hostname": "100.100.135.188",
"metadata_server": "etcd://100.100.135.188:2379;100.100.135.184:2379;100.100.135.174:2379",
"protocol": "ascend",
"master_server_address": "etcd://100.100.135.188:2379;100.100.135.184:2379;100.100.135.174:2379",
"global_segment_size": "16GB"
}
```

5. 在三个节点上启动vllm服务,拉起PD混部实例,--port改成各自的端口号:
```
export HCCL_INTRA_ROCE_ENABLE=1
export PYTHONPATH=$PYTHONPATH:/vllm-workspace/vllm-ascend
export PYTHONPATH=$PYTHONPATH:/vllm-workspace/vllm
export MOONCAKE_CONFIG_PATH=/data/liaobiting/mooncake/etcd/mooncake_config.json
export LD_LIBRARY_PATH=/usr/local/Ascend/cann-8.5.1/python/site-packages/mooncake:$LD_LIBRARY_PATH
export PYTHONHASHSEED=0
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export HCCL_OP_EXPANSION_MODE="AIV"
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=256
export ASCEND_AGGREGATE_ENABLE=1
export ACL_OP_INIT_MODE=1
export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000

vllm serve /data/0318-GLM-5-w4a8 \
--host 0.0.0.0 \
--port 30050 \
--data-parallel-size 2 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--seed 1024 \
--served-model-name glm-5 \
--max-model-len 8192 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--max-num-seqs 32 \
--quantization ascend \
--gpu-memory-utilization 0.95 \
--enforce-eager \
--enable-auto-tool-choice \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--speculative-config '{"num_speculative_tokens": 3, "method":"deepseek_mtp"}' \
--additional-config '{"enable_npugraph_ex": true, "fuse_qknorm_rope": true, "fuse_muls_add":true,"multistream_overlap_shared_expert":true,"recompute_scheduler_enable" : true}' \
--kv-transfer-config \
'{
"kv_connector": "AscendStoreConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"backend": "mooncake",
"lookup_rpc_port": "0",
"use_layerwise": false,
"load_async": true,
"register_buffer": true
}
}'
```

6. 查看master leader日志(当前leader为100.100.135.188节点),可以看到3个client都成功往master注册segment,mooncake pool申请的总内存为384.00 GB:
```
I0411 06:59:16.345165 21456 ha_helper.cpp:118] Init master service...
I0411 06:59:16.345239 21456 ha_helper.cpp:127] Init leader election helper...
I0411 06:59:16.345249 21456 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view
I0411 06:59:16.357301 21456 ha_helper.cpp:134] Trying to elect self as leader...
E0411 06:59:16.361210 21456 etcd_helper.cpp:52] key=mooncake-store/mooncake/master_view, error=key not found in etcd
I0411 06:59:16.361234 21456 ha_helper.cpp:57] No leader found, trying to elect self as leader
I0411 06:59:16.364097 21456 ha_helper.cpp:83] Successfully elected self as leader
I0411 06:59:21.364282 21456 ha_helper.cpp:154] Starting master service...
I0411 06:59:21.371884 21478 master_service.cpp:218] Task cleanup thread started
I0411 06:59:21.372367 21456 rpc_service.cpp:251] HTTP metrics server started on port 9003
I0411 06:59:21.373008 21480 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 0 B | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 0 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=0/0, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
I0411 07:12:24.980844 21488 master_service.cpp:140] client_id=13064254112467348141-14313097668496720285, action=mount_segment, segment_name=100.100.135.188:15525
I0411 07:12:24.986267 21487 master_service.cpp:140] client_id=9027241486787401593-10446720106337482151, action=mount_segment, segment_name=100.100.135.188:15041
I0411 07:12:25.011444 21487 master_service.cpp:140] client_id=4992189152740619462-13948519794634089884, action=mount_segment, segment_name=100.100.135.188:16024
I0411 07:12:25.018596 21489 master_service.cpp:140] client_id=6507021283039073625-2786715878051375291, action=mount_segment, segment_name=100.100.135.188:15538
I0411 07:12:25.021952 21489 master_service.cpp:140] client_id=5713498664078170643-10675103644390250166, action=mount_segment, segment_name=100.100.135.188:15574
I0411 07:12:25.054337 21487 master_service.cpp:140] client_id=5785571266135667182-664251338362233275, action=mount_segment, segment_name=100.100.135.188:15695
I0411 07:12:25.374733 21488 master_service.cpp:140] client_id=4634063046430273326-1201246848400043943, action=mount_segment, segment_name=100.100.135.188:16784
I0411 07:12:25.442077 21486 master_service.cpp:140] client_id=8594690077737356804-2335501872232384911, action=mount_segment, segment_name=100.100.135.188:15643
I0411 07:12:31.385146 21480 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 128.00 GB (0.0%) | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 8 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=64/64, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
I0411 07:12:35.137794 21487 master_service.cpp:140] client_id=14073381318776370857-12427840308647569558, action=mount_segment, segment_name=100.100.135.184:16127
I0411 07:12:35.138862 21486 master_service.cpp:140] client_id=10323742651643543980-837288166933932680, action=mount_segment, segment_name=100.100.135.184:15718
I0411 07:12:35.145296 21489 master_service.cpp:140] client_id=92697309230808826-13499916395937481603, action=mount_segment, segment_name=100.100.135.184:16910
I0411 07:12:35.148383 21488 master_service.cpp:140] client_id=10467344222303797334-15347246822976958393, action=mount_segment, segment_name=100.100.135.184:15924
I0411 07:12:35.151335 21487 master_service.cpp:140] client_id=2902955467842874137-3762847442476189575, action=mount_segment, segment_name=100.100.135.184:15295
I0411 07:12:35.153826 21489 master_service.cpp:140] client_id=12844949099844694567-5223621671726989990, action=mount_segment, segment_name=100.100.135.184:16170
I0411 07:12:35.164846 21488 master_service.cpp:140] client_id=3983622188728728208-16887971863470710701, action=mount_segment, segment_name=100.100.135.184:16869
I0411 07:12:35.172396 21489 master_service.cpp:140] client_id=813526925190136540-103126599260682900, action=mount_segment, segment_name=100.100.135.184:15804
I0411 07:12:41.385353 21480 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 256.00 GB (0.0%) | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 16 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=200/200, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
I0411 07:16:25.219340 21486 master_service.cpp:140] client_id=16018934717548921846-7017447522425649586, action=mount_segment, segment_name=100.100.135.174:15206
I0411 07:16:25.470141 21488 master_service.cpp:140] client_id=3406655645781847541-1729397808235232948, action=mount_segment, segment_name=100.100.135.174:16707
I0411 07:16:25.594974 21487 master_service.cpp:140] client_id=94039042344689938-15311935023949681042, action=mount_segment, segment_name=100.100.135.174:16724
I0411 07:16:25.624270 21486 master_service.cpp:140] client_id=9386941489814615425-7453681027279583876, action=mount_segment, segment_name=100.100.135.174:16122
I0411 07:16:25.985270 21489 master_service.cpp:140] client_id=14214988508567379209-10174589735368480415, action=mount_segment, segment_name=100.100.135.174:15493
I0411 07:16:26.163166 21487 master_service.cpp:140] client_id=15727232159109104122-12235209984021710515, action=mount_segment, segment_name=100.100.135.174:16711
I0411 07:16:31.276490 21488 master_service.cpp:140] client_id=10467041364011638163-288455838745867439, action=mount_segment, segment_name=100.100.135.174:16432
I0411 07:16:33.978780 21489 master_service.cpp:140] client_id=9675150528366022023-8683026668919070137, action=mount_segment, segment_name=100.100.135.174:16116
I0411 07:16:41.389412 21480 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 384.00 GB (0.0%) | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 24 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=4176/4176, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
```

7. 往master leader节点发送请求,此时master日志中内存占用为31.82 GB:
```
I0411 15:38:11.914714 21480 rpc_service.cpp:41] Master Metrics: Mem Storage: 31.82 GB / 384.00 GB (8.3%) | SSD Storage: 0 B / 0 B | Keys: 2400 (soft-pinned: 0) | Clients: 24 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=726180/726180, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=1236/0/1236, Item=2400/2400), PutEnd:(Req=1236/0/1236, Item=2400/2400), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=8/0/8, Item=8/8), ExistKey:(Req=1654/0/1654, Item=5084/5084), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
```

8. 手动kill master leader节点上mooncake_master进程:
```
pkill -f -9 mooncake_master
```

9. 100.100.135.184成为新的master leader,pool里的总内存384 GB和之前一样,但是pool里已使用的内存始终为0,没有看到client来mount_segment的日志:
```
I0411 07:00:00.743083 24321 ha_helper.cpp:42] CurrentLeader=100.100.135.188:50051, CurrentVersion=20
I0411 07:00:00.743108 24321 ha_helper.cpp:46] Waiting for leadership change...
I0411 15:39:21.213620 24321 ha_helper.cpp:83] Successfully elected self as leader
I0411 15:39:26.213907 24321 ha_helper.cpp:154] Starting master service...
I0411 15:39:26.221779 35383 master_service.cpp:218] Task cleanup thread started
I0411 15:39:26.222313 24321 rpc_service.cpp:251] HTTP metrics server started on port 9003
I0411 15:39:26.222890 35385 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 0 B | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 0 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=0/0, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
I0411 15:39:36.223074 35385 rpc_service.cpp:41] Master Metrics: Mem Storage: 0 B / 384.00 GB (0.0%) | SSD Storage: 0 B / 0 B | Keys: 0 (soft-pinned: 0) | Clients: 24 | Requests (Success/Total): PutStart=0/0, PutEnd=0/0, PutRevoke=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, Ping=224/224, CopyStart=0/0, CopyEnd=0/0, CopyRevoke=0/0, MoveStart=0/0, MoveEnd=0/0, MoveRevoke=0/0 | Batch Requests (Req=Success/PartialSuccess/Total, Item=Success/Total): PutStart:(Req=0/0/0, Item=0/0), PutEnd:(Req=0/0/0, Item=0/0), PutRevoke:(Req=0/0/0, Item=0/0), Get:(Req=0/0/0, Item=0/0), ExistKey:(Req=0/0/0, Item=0/0), QueryIp:(Req=0/0/0, Item=0/0), Clear:(Req=0/0/0, Item=0/0), CreateMoveTask:(Req=0/0), CreateCopyTask:(Req=0/0), QueryTask=(Req=0/0), FetchTasks=(Req=0/0), MarkTaskToComplete= (Req=0/0), | Eviction: Success/Attempts=0/0, keys=0, size=0 B | Discard: Released/Total=0/0, StagingSize=0 B
```

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the three-node HA setup with etcd, mooncake_master, and vLLM, then inspect the leader-election and metadata paths involved when the leader is killed. Compare the old and new leader logs and master-pool metrics before and after failover. Done means registered segments and metadata remain available after leadership changes instead of returning to zero.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.