pcm-memory can not reflect memory bandwidth accurately when using rdma (pcie to memory)
- Dominant language
- C++
- Stars
- 3.3k
- Forks
- 528
- Avg merge
- 1h 40m
- Merged PRs (30d)
- 2
Description
When doing ib_write_bw (a standard IB bandwidth testing tool without memory copy) and watch the corresponding memory bandwidth by pcm-memory ( of latest build 202502) as follows:
nodeA ip : 100.7.36.221, with ib port of mlx5_0 (400Gb) on numa0, and cpu is intel Xeon 6740E (BirthStream, Siera Forest)
nodeB ip : 100.7.36.223, with ib port of mlx5_0 (400Gb) on numa0, and cpu is intel Xeon 6740E (BirthStream, Siera Forest)
1. let nodeA do 'ib_write_bw' to nodeB:
nodeB(server): numactl -N 0 -m 0 ib_write_bw -d mlx5_0 -t 128 --run_infinitely
nodeA(client) : numactl -N 0 -m 0 ib_write_bw -d mlx5_0 -t 128 --run_infinitely 100.7.36.223
now ib_write_bw show the ib bandwidth is 44GB, and pcm-memory shows the memory bandwidth as 47GB, it's just ok.
2. let nodeB do 'ib_write_bw' to nodeA:
nodeA(server): numactl -N 0 -m 0 ib_write_bw -d mlx5_0 -t 128 --run_infinitely
nodeB(client) : numactl -N 0 -m 0 ib_write_bw -d mlx5_0 -t 128 --run_infinitely 100.7.36.221
now ib_write_bw show the ib bandwidth is also 44GB, but pcm-memory shows the memory bandwidth less than 1G, only a few hundred megabytes.


It looks like the pcm-memory does not account the memory band in last scenario 2.
Why? As i thought, the pcm-memory should account any memory bandwidth from cpu and pcie to memory ?
Contributor guide
Assessment
This issue has not been assessed yet.