[DOC] System Tuning Tutorial + tune_system.py description gaps for IGX Thor DevKit
@RamyaGuru is already working on this.
Since Aug 4, 2026.
- Dominant language
- C++
- Stars
- 36
- Forks
- 11
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 25
Description
Body
Three documentation gaps surfaced while running the raw GPUDirect sweep on an IGX Thor devkit (aarch64, driver 580.00/CUDA 13.0, integrated Thor iGPU sm_110 + discrete RTX PRO 6000 sm_120).
1. tune_system.py points to the wrong package for ibdev2netdev
tune_system.py warns users to run apt install infiniband-diags when ibdev2netdev is not found. On Ubuntu 24.04, infiniband-diags ships without ibdev2netdev — that script is part of mlnx-tools (or DOCA-Host), not infiniband-diags. Installing the suggested package leaves the tool in the same broken state, and the MRRS/MPS/MTU checks are silently skipped.
Fixes needed:
- Correct the hint to point at
mlnx-tools/ DOCA-Host (oribv_devinfofromibverbs-utilsas a fallback). - Ideally add a fallback that reads MRRS directly from PCIe config via
/sys/class/net/<dev>/device→lspci -vvs <bdf> | grep MaxReadReq(orsetpcion the DevCtl register) whenibdev2netdevis absent.
2. tune_system.py GPU checks always target index 0 — wrong on Tegra iGPU+dGPU systems
All GPU checks in tune_system.py (BAR1 size, GPUDirect-RDMA support, peermem, PIX-PXB topology) default to GPU index 0. On IGX Thor, index 0 is the integrated Thor iGPU (sm_110). The tool therefore reports "integrated GPU, no resizable BAR1, use kind: host_pinned, peermem N/A" — all of which is l target: the discrete RTX PRO 6000 (index 1), which has a real 256 MiB BAR1 and workingdma-buf GPUDirect.
Additional complexity — CUDA vs nvidia-smi index inversion on Tegra: the CUDA runtime enumerates GPUs in the opposite order from nvidia-smi/NVML on Tegra. With CUDA_VISIBLE_DEVICES unset, libcuda sonly the iGPU (count=1). Setting CUDA_VISIBLE_DEVICES=1 selects the iGPU (CUDA index 11 = dGPU — opposite). The only reliable selector is the UUID:CUDA_VISIBLE_DEVICES=GPU-<uuid>.
Fix needed: - Add a --gpu N flag (or auto-select the discrete GPU, or check every GPU) so the BAR1/GPUet the intended device.
- Document the Tegra CUDA-vs-nvidia-smi index inversion and recommend using the GPU UUID as the selector.
3. Docs gap: --privileged overrides --gpus '"device=N"' on Tegra; no guidance on GPU pinning
The documented container launch uses --privileged (required for DPDK). On a system with both an integrated and a discrete NVIDIA GPU, --privileged floods /dev with all Tegra iGPU nodes, which causes the CUDA runtime to enumerate only the iGPU regardless of --gpus '"device=N"'. affinity and wrong device.
The reliable workaround is -e CUDA_VISIBLE_DEVICES=GPU-<uuid> (by UUID, not by index — seeersion).
Docs to update:
docs/benchmarks/raw_benchmarking.mddocs/tutorials/system_configuration.md
Add a "Tegra / IGX with a discrete GPU" note: when running with --privileged on a system t dGPU, pin the target GPU with CUDA_VISIBLE_DEVICES=GPU-<uuid> rather than by index.
Discovered on: IGX Thor aarch64, kernel 6.8.0-1019-nvidia-tegra-rt, host driver 580.00oceP4p3s0f0/f1), daqiri 1134167`.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.