CUPTI metrics not recorded in profiler
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
### 🐛 Describe the bug
When recording traces with CUPTI counters enabled (i.e. `--with-cupti`), there are two issues that prevent me from using the resulting traces:
- The resulting traces don't seem to include any CUPTI metrics
- The resulting traces include a lot of invalid spans (e.g. containing `"ts": 0.000`)
A pair of sample traces with/without CUPTI can be found in [this gist](https://gist.github.com/kwohlfahrt/479fc10a886a8cdddf2f805620805ff5). They were generated by running the script below in the `pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel` Docker image, once with profiling metrics enabled (`--with-cupti`) and once without.
It doesn't make a difference whether `profiler_measure_per_kernel` is enabled or not.
```python3
import sys
import torch
from torch.profiler import profile, ProfilerActivity, schedule, _ExperimentalConfig
model = torch.nn.Sequential(*[torch.nn.Linear(128, 128) for _ in range(5)]).to("cuda")
opt = torch.optim.Adam(model.parameters())
profiler_metrics = []
if "--with-cupti" in sys.argv:
profiler_metrics.extend([
"kineto__tensor_core_insts",
"dram__bytes_read.sum",
"dram__bytes_write.sum",
])
prof_schedule = schedule(skip_first=5, wait=1, warmup=1, active=1, skip_first_wait=True)
activities = [ProfilerActivity.CPU, ProfilerActivity.CUDA]
with profile(
activities=activities,
schedule=prof_schedule,
experimental_config=_ExperimentalConfig(profiler_metrics=profiler_metrics),
record_shapes=True,
) as prof:
for x in range(10):
opt.zero_grad()
x = torch.randn((2, 128), device="cuda")
output = model(x)
loss = output.sum()
loss.backward()
opt.step()
prof.step()
if "--with-cupti" in sys.argv:
prof.export_chrome_trace("trace-cupti.json")
else:
prof.export_chrome_trace("trace.json")
```
### Versions
Note - this is running in the `pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel` Docker image.
```
Collecting environment information...
PyTorch version: 2.9.1+cu128
Is debug build: False
CUDA used to build PyTorch: 12.8
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 4.1.2
Libc version: glibc-2.35
Python version: 3.11.14 | packaged by conda-forge | (main, Oct 22 2025, 22:46:25) [GCC 14.3.0] (64-bit runtime)
Python platform: Linux-5.14.0-611.9.1+2.1.el9_7_ciq.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.8.93
CUDA_MODULE_LOADING set to:
GPU models and configuration: GPU 0: NVIDIA L4
Nvidia driver version: 590.44.01
cuDNN version: Could not collect
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Caching allocator config: N/A
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
Stepping: 7
BogoMIPS: 4400.43
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 64 KiB (2 instances)
L1i cache: 64 KiB (2 instances)
L2 cache: 2 MiB (2 instances)
L3 cache: 38.5 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-3
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Mitigation; Aligned branch/return thunks
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Vmscape: Not affected
Versions of relevant libraries:
[pip3] numpy==2.3.4
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] optree==0.17.0
[pip3] torch==2.9.1+cu128
[pip3] torchaudio==2.9.1+cu128
[pip3] torchelastic==0.2.2
[pip3] torchvision==0.24.1+cu128
[pip3] triton==3.5.1
[conda] numpy 2.3.4 py311h2e04523_0 conda-forge
[conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.27.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi
[conda] optree 0.17.0 pypi_0 pypi
[conda] torch 2.9.1+cu128 pypi_0 pypi
[conda] torchaudio 2.9.1+cu128 pypi_0 pypi
[conda] torchelastic 0.2.2 pypi_0 pypi
[conda] torchvision 0.24.1+cu128 pypi_0 pypi
[conda] triton 3.5.1 pypi_0 pypi
```
cc @robieta @chaekit @guotuofeng @guyang3532 @dzhulgakov @davidberard98 @briancoutinho @sraikund16 @sanrise @mwootton
Contributor guide
Assessment
This issue has not been assessed yet.