intel / intel/pcm

pcm-memory can not reflect memory bandwidth accurately when using rdma (pcie to memory)

Open
#923 3 comments 0 reactions 0 assignees View on GitHub
invalid question
Dominant language
C++
Stars
3.3k
Forks
528
Avg merge
1h 40m
Merged PRs (30d)
2

Description

When doing ib_write_bw (a standard IB bandwidth testing tool without memory copy) and watch the corresponding memory bandwidth by pcm-memory ( of latest build 202502) as follows:
nodeA ip : 100.7.36.221, with ib port of mlx5_0 (400Gb) on numa0, and cpu is intel Xeon 6740E (BirthStream, Siera Forest)
nodeB ip : 100.7.36.223, with ib port of mlx5_0 (400Gb) on numa0, and cpu is intel Xeon 6740E (BirthStream, Siera Forest)
1. let nodeA do 'ib_write_bw' to nodeB:
nodeB(server): numactl -N 0 -m 0 ib_write_bw -d mlx5_0 -t 128 --run_infinitely
nodeA(client) : numactl -N 0 -m 0 ib_write_bw -d mlx5_0 -t 128 --run_infinitely 100.7.36.223
now ib_write_bw show the ib bandwidth is 44GB, and pcm-memory shows the memory bandwidth as 47GB, it's just ok.
2. let nodeB do 'ib_write_bw' to nodeA:
nodeA(server): numactl -N 0 -m 0 ib_write_bw -d mlx5_0 -t 128 --run_infinitely
nodeB(client) : numactl -N 0 -m 0 ib_write_bw -d mlx5_0 -t 128 --run_infinitely 100.7.36.221
now ib_write_bw show the ib bandwidth is also 44GB, but pcm-memory shows the memory bandwidth less than 1G, only a few hundred megabytes.

![Image](https://github.com/user-attachments/assets/b0f249b7-9906-4815-a23b-4e122d5bddda)
![Image](https://github.com/user-attachments/assets/e9756216-ef3a-4987-be36-650138225524)

It looks like the pcm-memory does not account the memory band in last scenario 2.
Why? As i thought, the pcm-memory should account any memory bandwidth from cpu and pcie to memory ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.