kvcache-ai / kvcache-ai/Mooncake
[Feature Request]: Mooncake连接NVMe-of SSD
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Describe your feature request
引言
Mooncake的nvme of硬盘挂载方案是为了将KVCache卸载到远端SSD上,以绿算GP5000作为NVMe-of设备挂载到Mooncake的实现案例。
具体步骤
先使用SPDK创建NVMe-of硬盘,拿到nqn信息再注册到Mooncake内部,而不能直接通过nqn信息和IP进行注册,现在通过调用Mooncake接口信息能通过ip和nqn直接注册到Mooncake, 从而省去了使用SPDK创建NVMe-of硬盘这一步,希望后续官方能把这个功能直接集成。
绿算GP设备介绍:GP 系列通用直连数据栈是面向大模型推理 KV Cache 场景打造的硬件级 NVMe-oF 存储设备。设备可构建集群全局共享缓存池,打通跨节点缓存互通,KV Cache 跨机读取端到端延迟≤20μs;第一代GP-5000具备 1620 万 IOPS 并发能力,自动打散热点数据,解决本地 SSD 方案单点 IO 过载带来的长尾延迟。同时支持计算节点 Diskless 无盘部署,省去服务器本地 NVMe 盘,降低硬件运维成本与功耗,整体集群 TCO 可降低 30% 以上,适配 Mooncake NVMe-oF 远端存储架构落地。
1、配置节点IP
Mooncake节点需要有支持RDMA的网卡,IP需要和绿算技术GP设备网络互通,例如GP设备IP:10.10.10.100,节点IP:10.10.10.200
2、节点设置hugepages,
echo "vm.nr_hugepages=4096" > /etc/sysctl.d/spdk.conf && sysctl -p /etc/sysctl.d/spdk.conf
sysctl -a | grep hugepages
3、根据官网案例编译Mooncake
sudo bash dependencies.sh
mkdir build
cd build
cmake .. -DUSE_NOF=ON
make -j
sudo make install
4、获取绿算技术GP5000的nqn
nvme discover -t rdma -a 10.10.10.100
5、使用spdk作为启动器挂载NVMe-of ssd
nohup build/bin/spdk_tgt -m 0x3 -s 4096 &
# 此时查看返回为空
./scripts/rpc.py bdev_get_bdevs
./scripts/rpc.py bdev_nvme_attach_controller -b nvme1 -t rdma -a 10.10.10.100 -f ipv4 -s 4420 -n nqn.ForinnBase5000.lsjs:nvme.1
./scripts/rpc.py bdev_nvme_attach_controller -b nvme3 -t rdma -a 10.10.10.100 -f ipv4 -s 4420 -n nqn.ForinnBase5000.lsjs:nvme.3
./scripts/rpc.py bdev_nvme_attach_controller -b nvme5 -t rdma -a 10.10.10.100 -f ipv4 -s 4420 -n nqn.ForinnBase5000.lsjs:nvme.5
./scripts/rpc.py bdev_nvme_attach_controller -b nvme7 -t rdma -a 10.10.10.100 -f ipv4 -s 4420 -n nqn.ForinnBase5000.lsjs:nvme.7
# 再次查看即可看到nvme1n1 nvme3n1 nvme5n1 nvme7n1
./scripts/rpc.py bdev_get_bdevs
6、启动Mooncake master
mooncake_master --rpc_address=0.0.0.0 --rpc_port=50051 \
--enable_http_metadata_server=true \
--http_metadata_server_port=8080 \
--http_metadata_server_host=0.0.0.0 \
--nof_heartbeat_interval_sec=86400 \
--nof_heartbeat_probe_timeout_ms=60000 \
--nof_heartbeat_failures_threshold=999999
7、注册NVMe-of ssd
单个注册
python3 /opt/mooncake/nvmf_register.py \
--master 10.10.10.200:50051 \
--nqn nqn.ForinnBase5000.lsjs:nvme.1 \
--traddr 10.10.10.100 --trsvcid 4420 \
--spdk-rpc /opt/spdk/spdk-master/scripts/rpc.py
多个注册
python3 /opt/mooncake/nvmf_register.py \
--master 10.10.10.200:50051 \
--nqn nqn.ForinnBase5000.lsjs:nvme.1 \
--nqn nqn.ForinnBase5000.lsjs:nvme.3 \
--nqn nqn.ForinnBase5000.lsjs:nvme.5 \
--nqn nqn.ForinnBase5000.lsjs:nvme.7 \
--traddr 10.10.10.100 --trsvcid 4420 \
--spdk-rpc /opt/spdk/spdk-master/scripts/rpc.py
注册完成可以在master日志中看到 NVMe-of 硬盘容量
注册成功日志图片
master日志图片
注意:Mooncake有心跳检测机制,每次检测都会初始化SPDK,但是SPDK只能初始化一次,master服务启动必须加入以下参数,绕过Mooncake心跳检测
--nof_heartbeat_interval_sec=86400 \
--nof_heartbeat_probe_timeout_ms=60000 \
--nof_heartbeat_failures_threshold=999999
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Research direction
Start with nvmf_register.py and trace the Mooncake master registration interface it invokes. Reproduce the documented GP5000 setup and current IP/NQN registration flow, then identify how registration currently depends on SPDK. Done means direct IP/NQN registration works without the manual SPDK setup and the master logs show the NVMe-oF capacity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux, python
- Domain
- backend, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100