kvcache-ai / kvcache-ai/Mooncake
[Bug]: NoF 写入失败:`spdk_rdma_get_translation: No translation for ptr` 是否是我配置的不对导致的这个问题?
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
### 问题描述
当 Mooncake Store 客户端使用 `ReplicateConfig{replica_num=0, nof_replica_num=1}`(仅 NoF 副本模式)写入对象时,Master 成功在 NVMe-oF SSD 池上分配空间,SPDK NVMe-oF controller 也成功 attach。但实际数据传输失败,因为 SPDK 的 RDMA 传输层找不到用户 buffer 的内存翻译记录。
### 环境信息
- Mooncake 版本:v0.3.12(tag `v0.3.12`,commit `a1ca71f4`)
- 同时验证了 `v0.3.12-pre1-185-ga1ca71f4`(截至 2026-07-27 的最新 main 分支)——**代码完全相同,未修复**
- SPDK:v23.01.1
- RDMA 网卡:ConnectX-6(mlx5_0),RoCE v2
- NVMe-oF target:Samsung PM1753 over RDMA(192.168.3.110:4420-4423)
- LMCache:v0.5.3.dev30(MP 模式,`lmcache server` + `mooncake_store` L2 adapter)
### 复现步骤
1. 启动 Mooncake master + metadata server,注册 NVMe-oF SSD 池(4 × Samsung PM1753 namespace)
2. 启动 Mooncake Store 客户端,配置 `protocol: "rdma"`、`global_segment_size: "0"`、`local_buffer_size: 536870912`
3. 调用 `batch_put_from(key, buffer, size, ReplicateConfig{replica_num=0, nof_replica_num=1})`,其中 `buffer` 指向已通过 `client->register_buffer()` 预注册的 DRAM 内存
### 期望行为
对象数据通过 SPDK RDMA 写入 NVMe-oF SSD 池。
### 实际行为
Master 日志显示分配成功:
```
master_service.cpp:3088] key=..., config=ReplicateConfig: { replica_num: 0, nof_replica_num: 1, ... }, action=put_start_begin
allocator.cpp:261] allocation_succeeded size=33554432 segment=traddr:192.168.3.110 trsvcid:4422 subnqn:nqn.1994-11.com.samsung:nvme:PM1753:2.5-inch:S8FFNG0Y901187 trtype:RDMA
```
SPDK NVMe-oF controller attach 成功:
```
spdk_wrapper.cpp:297] Attached to 192.168.3.110 nqn.1994-11.com.samsung:nvme:PM1753:2.5-inch:S8FFNG0Y901187
```
但数据传输失败:
```
common.c:216: spdk_rdma_get_translation: *ERROR*: No translation for ptr 0x7f1644000000, size 131072
nvme_rdma.c:1461: nvme_rdma_get_memory_translation: *ERROR*: RDMA memory translation failed, rc -22
nvme_rdma.c:2316: nvme_rdma_qpair_submit_request: *ERROR*: nvme_rdma_req_init() failed
transfer_task.cpp:511] work 0, seg 0x7f1398016730 submit io fail
client_service.cpp:2449] Operation for key ... failed: TRANSFER_FAIL (Transfer 0 failed)
```
`ptr 0x7f1644000000` 是用户提供的 buffer(LMCache L1 DRAM,128KB chunk = `MC_NOF_SUBMIT_CHUNK_BYTES`)。该 buffer 已通过 Mooncake Transfer Engine 的 `client->register_buffer()` → `ibv_reg_mr()` 注册,但**未注册到 SPDK NVMe-oF RDMA 的内存映射**。
### 根因分析
`transfer_task.cpp` 的 NoF 写入路径直接将用户 buffer 指针传给 SPDK,没有先注册到 SPDK NVMe-oF RDMA 传输层的内存映射:
```cpp
// transfer_task.cpp:491-503(SpdkNofWorkerPool::workerThread)
void* submit_ptr = reinterpret_cast(
reinterpret_cast(task->ptr) + lba_off * block_size);
int ret = SpdkWrapper::GetInstance().SubmitRequest(
task->seg_handle, submit_ptr, submit_lba, submit_lba_count, ...);
```
```cpp
// spdk_wrapper.cpp:387(SpdkWrapper::SubmitRequest)
return spdk_nvme_ns_cmd_write(ns, qpair, ptr, lba, lba_count, cb_fn, cb_ctx, 0);
```
SPDK 的 NVMe-oF RDMA 传输层维护着自己独立的 RDMA 内存映射(`nvme_rdma_qpair::rkey_map`),与 Mooncake Transfer Engine 的 RDMA 内存注册**相互独立**。当 `spdk_nvme_ns_cmd_write()` 被调用时,SPDK 内部通过 `nvme_rdma_get_memory_translation()` 在自己的内存映射中查找 buffer。由于用户 buffer 从未注册到 SPDK 的 RDMA context,查找失败返回 `-22 (EINVAL)`。
### Before submitting...
- [ ] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Start in transfer_task.cpp at SpdkNofWorkerPool::workerThread and trace the buffer passed to SpdkWrapper::SubmitRequest in spdk_wrapper.cpp. Compare Mooncake's client-register_buffer registration with SPDK's nvme_rdma memory-translation lookup, then reproduce the NoF-only batch_put_from case. Done means the registered user buffer passes SPDK translation and the NVMe-oF write completes without TRANSFER_FAIL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100