OWRS Performance Enhancement
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 35
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
I ran smart's code on my testbed, which brought a huge performance boost. My testbed uses a connectx-6 NIC and two Intel(R) Xeon(R) Gold 5218 CPUs.
When I wanted to replicate the performance gains of owrs, I wrote my own test code, which just posted depth wrs and poll all. But I couldn't get the same performance gain with my code.
Here are the throughputs at 8byte using smart and my test code.
I turned off all optimization options for smart except thread_aware_alloc. And made my test code as close as possible to the qp optimization and owrs optimization that smart uses. But no matter what, I can't get similar performance improvement above 24 threads and above 8 depth. Can you give me some idea about the source of the performance improvement in smart. Here is the smart_config I am using.
Also, my testing found that qp's allocation optimization on the doorbell register is not applied above 12 (which is the actual driver limit) (I turned off preload), but the smart code still gets a higher performance boost with more shared_uuar than 12. This is something I can't understand either.
{
"infiniband": {
"name": "",
"port": 1,
"gid_idx": 1
},
"qp_param": {
"max_cqe_size": 256,
"max_wqe_size": 256,
"max_sge_size": 1,
"max_inline_data": 64
},
"max_nodes": 128,
"initiator_cache_size": 4096,
"use_thread_aware_alloc": true,
"thread_aware_alloc": {
"total_uuar": 100,
"shared_uuar": 96,
"shared_cq": true
},
"use_work_req_throt": false,
"work_req_throt": {
"initial_credit": 4,
"max_credit": 12,
"credit_step": 2,
"execution_epochs": 60,
"sample_cycles": 19200000,
"inf_credit_weight": 1.05,
"auto_tuning": false
},
"use_conflict_avoidance": false,
"use_speculative_lookup": false,
"experimental": {
"qp_sharing": false
}
}
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by comparing the smart configuration and the custom post-depth-WR/poll-all test, focusing on thread_aware_alloc, shared_uuar, shared_cq, qp optimization, and owrs optimization. Done means identifying and documenting the source of the throughput difference, including why performance changes above 24 threads and 8 depth and why shared_uuar above 12 still helps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100