NVIDIA / NVIDIA/open-gpu-kernel-modules
Expose globally identifiable per-SM/SMSP performance samples
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 17.4k
- Forks
- 1.9k
- PR merge metrics
- No merged PRs in 30d
Description
Summary
There is currently no supported NVIDIA interface I could find that exposes time-resolved performance-monitor samples for each globally identifiable physical SM/SMSP instance, for example:
<physical SM ID, SMSP ID, timestamp, activity>
Related Ideas discussion: https://github.com/NVIDIA/open-gpu-kernel-modules/discussions/1350
The existing tools provide valuable but incomplete views:
nvidia-smireports aggregate GPU utilization.- Nsight Compute
sm__*/smsp__*metrics generally expose rollups such as sum/average/min/max. smsp__warpidsamp_*exposes documented<SMSP ID>:<Warp ID>instance keys, but no physical SM coordinate. On my test system, output such as0:0,0:4,0:8, and0:12was useful for scheduler research but could not identify 680 physical SMSPs across the chip.- CUPTI/PM sampling provides a timeline, but I could not find a supported way to retain a globally identifiable physical SM/SMSP coordinate for every sample.
- PTX exposes
%smidand%warpid, but no%smspid; in-kernel instrumentation is intrusive and is not equivalent to reading performance-monitor utilization.
Reproducer / motivating experiment
Project: https://github.com/Loong0x00/rtx5090-smsp-badapple
An RTX 5090 has 170 enabled SMs and four SMSPs per SM: 170 × 4 = 680 = 34 × 20. The project launches 170 cooperative CTAs, limits occupancy to one CTA per SM, and uses four warps per CTA to encode a 34×20 binary animation as FP32 issue activity versus nanosleep.
The included placement and contention probes establish:
- simultaneous sampled coverage of 170/170 SMs;
- four distinct physical warp slots per CTA;
- on the tested GB202, warps 0/1/2/3 do not contend while 0/4/8/12 do, supporting the empirical formula
SMSP = physical_warpid & 3for this exact stack.
However, the experiment cannot observe the resulting 680-channel physical activity pattern through a supported NVIDIA monitor. The visible output has to be reconstructed through application instrumentation, which is precisely what the requested monitor would avoid.
Research notes, including PTX/SASS special-register probes and Nsight Compute sampling output:
https://github.com/Loong0x00/rtx5090-smsp-badapple/blob/main/research/SM120_HIDDEN_REGISTERS.md
Requested interface
Could NVIDIA expose an authorized, documented profiler/CUPTI mode that provides:
- a stable per-capture physical SM instance ID;
- the SMSP/scheduler index within that SM;
- timestamped activity, issued-instruction, active-warp, or programmable-trigger samples;
- clear documentation of aggregation, sampling, replay, and identifier lifetime;
- an offline/privileged mode if unrestricted access would create performance-counter security concerns?
This does not need to be a CUDA programming-model affinity guarantee, nor does it need to permit warp placement control. A profiler-only observation interface would be sufficient.
If this belongs to the user-mode Nsight Compute/CUPTI stack rather than the open kernel modules, please point to a public tracker that accepts this request.
Environment
- GPU: NVIDIA GeForce RTX 5090 / GB202, compute capability 12.0
- Driver: 610.57.04
- CUDA: 13.3
- Nsight Compute: 2026.2.1
- OS: Arch Linux, kernel 7.2.3-arch1-2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by determining whether the requested profiler/CUPTI functionality belongs in this repository or the user-mode Nsight stack, using the linked Related Ideas discussion. Review research/SM120_HIDDEN_REGISTERS.md and the existing kernel-module interfaces mentioned in the issue; done would require a defined implementation path for stable SM/SMSP identifiers and timestamped samples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100