dmlc / dmlc/ps-lite

Why not using RDMA WRITE( with imm) or RDMA READ to avoid additional memory copy?

Open
#174 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
1.6k
Forks
540
PR merge metrics
No merged PRs in 30d

Description

Hi developers of dmlc.
Recently I've been working with ps-lite. I find out that in the operation of `KVWorker::Pull`, there maybe some memory copy operations that may cause performance overhead:
```
template
void KVServer::Response(const KVMeta& req, const KVPairs& res) {
Message msg;
msg.meta.app_id = obj_->app_id();
msg.meta.customer_id = req.customer_id;
msg.meta.request = false;
msg.meta.push = req.push;
msg.meta.pull = req.pull;
msg.meta.head = req.cmd;
msg.meta.timestamp = req.timestamp;
msg.meta.recver = req.sender;
if (res.keys.size()) {
msg.AddData(res.keys);
msg.AddData(res.vals);
if (res.lens.size()) {
msg.AddData(res.lens);
}
}
Postoffice::Get()->van()->Send(msg);
}
```
In my opinion, code ` msg.AddData` holds back the communication between woker and server.
If using RDMA WRITE( with imm) or RDMA READ, we can exchange the address by out-of-band communication in the beginning, then do PULL and PUSH on these addresses.
This is just my thought. Please correct me if I'm wrong or I miss some details.

By the way, the function `Van::PackMetaPB` used by IBVERBS misses `pb->set_pull(meta.pull);` which is in the function of `Van::PackMeta` used by ZMQ. This will cause the error when running the test case `ps-lite/tests/test_kv_app.cc` using IBVERBS.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.