NVIDIA / NVIDIA/open-gpu-kernel-modules

Expose globally identifiable per-SM/SMSP performance samples

Open
#1,351 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
17.4k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

Summary

There is currently no supported NVIDIA interface I could find that exposes time-resolved performance-monitor samples for each globally identifiable physical SM/SMSP instance, for example:

<physical SM ID, SMSP ID, timestamp, activity>

Related Ideas discussion: https://github.com/NVIDIA/open-gpu-kernel-modules/discussions/1350

The existing tools provide valuable but incomplete views:

  • nvidia-smi reports aggregate GPU utilization.
  • Nsight Compute sm__* / smsp__* metrics generally expose rollups such as sum/average/min/max.
  • smsp__warpidsamp_* exposes documented <SMSP ID>:<Warp ID> instance keys, but no physical SM coordinate. On my test system, output such as 0:0, 0:4, 0:8, and 0:12 was useful for scheduler research but could not identify 680 physical SMSPs across the chip.
  • CUPTI/PM sampling provides a timeline, but I could not find a supported way to retain a globally identifiable physical SM/SMSP coordinate for every sample.
  • PTX exposes %smid and %warpid, but no %smspid; in-kernel instrumentation is intrusive and is not equivalent to reading performance-monitor utilization.
Reproducer / motivating experiment

Project: https://github.com/Loong0x00/rtx5090-smsp-badapple

An RTX 5090 has 170 enabled SMs and four SMSPs per SM: 170 × 4 = 680 = 34 × 20. The project launches 170 cooperative CTAs, limits occupancy to one CTA per SM, and uses four warps per CTA to encode a 34×20 binary animation as FP32 issue activity versus nanosleep.

The included placement and contention probes establish:

  • simultaneous sampled coverage of 170/170 SMs;
  • four distinct physical warp slots per CTA;
  • on the tested GB202, warps 0/1/2/3 do not contend while 0/4/8/12 do, supporting the empirical formula SMSP = physical_warpid & 3 for this exact stack.

However, the experiment cannot observe the resulting 680-channel physical activity pattern through a supported NVIDIA monitor. The visible output has to be reconstructed through application instrumentation, which is precisely what the requested monitor would avoid.

Research notes, including PTX/SASS special-register probes and Nsight Compute sampling output:

https://github.com/Loong0x00/rtx5090-smsp-badapple/blob/main/research/SM120_HIDDEN_REGISTERS.md

Requested interface

Could NVIDIA expose an authorized, documented profiler/CUPTI mode that provides:

  1. a stable per-capture physical SM instance ID;
  2. the SMSP/scheduler index within that SM;
  3. timestamped activity, issued-instruction, active-warp, or programmable-trigger samples;
  4. clear documentation of aggregation, sampling, replay, and identifier lifetime;
  5. an offline/privileged mode if unrestricted access would create performance-counter security concerns?

This does not need to be a CUDA programming-model affinity guarantee, nor does it need to permit warp placement control. A profiler-only observation interface would be sufficient.

If this belongs to the user-mode Nsight Compute/CUPTI stack rather than the open kernel modules, please point to a public tracker that accepts this request.

Environment
  • GPU: NVIDIA GeForce RTX 5090 / GB202, compute capability 12.0
  • Driver: 610.57.04
  • CUDA: 13.3
  • Nsight Compute: 2026.2.1
  • OS: Arch Linux, kernel 7.2.3-arch1-2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by determining whether the requested profiler/CUPTI functionality belongs in this repository or the user-mode Nsight stack, using the linked Related Ideas discussion. Review research/SM120_HIDDEN_REGISTERS.md and the existing kernel-module interfaces mentioned in the issue; done would require a defined implementation path for stable SM/SMSP identifiers and timestamped samples.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.