NVIDIA / NVIDIA/cuda-samples

P2P is supported between GPUs that belong to different NUMA nodes

Open
#284 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
C++
Stars
9.6k
Forks
2.4k
Avg merge
53m
Merged PRs (30d)
1

Description

I've execute the simpleP2P on my server (GPU-L4, with the following topology) which GPU0 and GPU1 belongs to different numa nodes.

	GPU0	GPU1	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	SYS	0-7,32-39	0		N/A
GPU1	SYS	 X 	16-23,48-55	2		N/A

And I obtained the following results. Why is that? How does it support P2P (Peer-to-Peer) connections between GPUs that are on different PCIe root complexes?

./simpleP2P 
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA L4 (GPU0) -> NVIDIA L4 (GPU1) : Yes
> Peer access from NVIDIA L4 (GPU1) -> NVIDIA L4 (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
Creating event handles...
cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 19.87GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Disabling peer access...
Shutting down...
Test passed

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the simpleP2P sample and its reported peer-access and transfer results, then review the CUDA sample documentation or relevant topology behavior. Done means documenting why GPUs on different NUMA nodes or PCIe root complexes can report peer access and what the displayed transfer path implies.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.