cudaErrorLaunchOutOfResources in CTC loss backward on RTX 5090 (sm_120, Blackwell) with CUDA 13.0

### 🐛 Describe the bug

## Description

CTC loss backward raises `cudaErrorLaunchOutOfResources` on RTX 5090 (Blackwell, sm_120) with CUDA 13.0 when batch size × transcript length exceeds a certain threshold. The error occurs in `Variable._execution_engine.run_backward` and cannot be caught with `CUDA_LAUNCH_BLOCKING=1` at the Python level.

## Environment

- **GPU:** NVIDIA GeForce RTX 5090
- **Compute Capability:** sm_120 (Blackwell)
- **Driver Version:** 580.105.08
- **CUDA Version:** 13.0
- **PyTorch Version:** 2.9.0a0+50eac811a6.nv25.09
- **Python Version:** 3.12
- **OS:** Ubuntu (Docker container)

## Reproduction

```python
import torch

device = "cuda"
V = 3350
ctc_loss = torch.nn.CTCLoss(blank=0, reduction='none', zero_infinity=True)

# Failing case
B, T_enc, tlen = 8, 250, 65
encoded = torch.randn(B, T_enc, V + 1, device=device, requires_grad=True)
log_probs = encoded.log_softmax(dim=-1)
targets = torch.randint(1, V, (B, tlen), device=device)
encoded_len = torch.full((B,), T_enc, dtype=torch.long, device=device)
target_lengths = torch.full((B,), tlen, dtype=torch.long, device=device)

loss = ctc_loss(log_probs.permute(1, 0, 2), targets, encoded_len, target_lengths)
loss.mean().backward() # <-- cudaErrorLaunchOutOfResources here
```

## Threshold observed

Through systematic testing, the following threshold was identified:

| Batch size (B) | Max transcript length (OK) | Min transcript length (FAIL) |
|----------------|---------------------------|------------------------------|
| 1–3 | 173+ ✅ | (not found) |
| 4–7 | 120 ✅ | 130 ❌ |
| ≥ 8 | 60 ✅ | 65 ❌ |

The threshold appears to correspond to `next_power_of_2(2 × transcript_len + 1)`:
- transcript ≤ 120 → 241 → **256 threads** ✅
- transcript ≥ 128 → 257 → **512 threads** ❌ (exceeds register limit on sm_120)

## Key observations

- The error occurs **only during backward**, not forward
- `encoder backward` alone passes without error
- `CTC loss backward` is the failing component
- Reproducible with both `bf16` and `fp32` precision
- Reproducible with both single GPU and multi-GPU (DDP)
- Batch size alone does not trigger the error; transcript length is the primary factor

## Error message

```
torch.AcceleratorError: CUDA error: too many resources requested for launch
Search for `cudaErrorLaunchOutOfResources` in
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html
for more information.
```

## GPU resource limits

```
GPU: NVIDIA GeForce RTX 5090
Compute Capability: 12.0
Total Memory: 31.3 GB
Warp size: 32
Multi-processor count: 170
Max threads per multiprocessor: 1536
Regs per multiprocessor: 65536
→ Max registers per thread: 65536 / 1536 = 42.67
Shared memory per block: 49152 bytes
```

## Expected behavior

CTC loss backward should run successfully on sm_120 (Blackwell) with the same configurations that work on previous GPU architectures.

## Workaround

Limiting batch size to ≤ 3 avoids the error but severely degrades training throughput.

### Versions

`curl -sL https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py | python`

Result
```
Collecting environment information...
PyTorch version: 2.9.0a0+50eac811a6.nv25.09
Is debug build: False
CUDA used to build PyTorch: 13.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.3 LTS (x86_64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.39

Python version: 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-6.14.0-37-generic-x86_64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: 13.0.88
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 5090
Nvidia driver version: 580.105.08
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.13.1
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Caching allocator config: {'PYTORCH_CUDA_ALLOC_CONF': 'expandable_segments:True,max_split_size_mb:512'
}

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 7 9700X 8-Core Processor
CPU family: 26
Model: 68
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 0
Frequency boost: enabled
CPU(s) scaling MHz: 57%
CPU max MHz: 5582.0000
CPU min MHz: 600.0000
BogoMIPS: 7585.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_
good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor sss
e3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_le
gacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext
perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmc
all fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap av
x512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_
occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru w
bnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilt
er pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi
2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid bus_lock_detect movdiri movdir
64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d amd_lbr_pmc_freeze
Virtualization: AMD-V
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 8 MiB (8 instances)
L3 cache: 32 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Ghostwrite: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; IBPB on VMEXIT only
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanit
ization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIB
P always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Mitigation; IBPB on VMEXIT

Versions of relevant libraries:
[pip3] intel-openmp==2021.4.0
[pip3] mkl==2021.1.1
[pip3] mkl-devel==2021.1.1
[pip3] mkl-include==2021.1.1
[pip3] mypy_extensions==1.1.0
[pip3] numpy==1.26.4
[pip3] nv-one-logger-pytorch-lightning-integration==2.3.1
[pip3] nvidia-cuda-nvrtc==13.1.80
[pip3] nvidia-cuda-runtime==13.0.88
[pip3] nvidia-cudnn-frontend==1.16.0
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-pytriton==0.7.0
[pip3] nvtx==0.2.14
[pip3] onnx==1.18.0
[pip3] onnx_graphsurgeon==0.5.8
[pip3] onnx-ir==0.1.12
[pip3] onnxscript==0.5.7.dev20251209
[pip3] open_clip_torch==3.2.0
[pip3] optree==0.17.0
[pip3] pytorch-lightning==2.6.0
[pip3] tbb==2021.13.1
[pip3] torch==2.9.0a0+50eac811a6.nv25.9
[pip3] torch_tensorrt==2.9.0a0
[pip3] torchao==0.13.0+git
[pip3] torchdiffeq==0.2.5
[pip3] torchmetrics==1.8.2
[pip3] torchprofile==0.0.4
[pip3] torchsde==0.2.6
[pip3] torchvision==0.24.0a0+98f8b375
[pip3] torchx==0.7.0
[pip3] triton==3.4.0+gitc817b9b6
[pip3] tritonclient==2.51.0
[conda] Could not collect
```

```
curl -OL https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py
#For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

Result
```
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 31107 100 31107 0 0 1051k 0 --:--:-- --:--:-- --:--:-- 1084k
Collecting environment information...
PyTorch version: 2.9.0a0+50eac811a6.nv25.09
Is debug build: False
CUDA used to build PyTorch: 13.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.3 LTS (x86_64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.39

Python version: 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-6.14.0-37-generic-x86_64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: 13.0.88
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 5090
Nvidia driver version: 580.105.08
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.13.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.13.1
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Caching allocator config: {'PYTORCH_CUDA_ALLOC_CONF': 'expandable_segments:True,max_split_size_mb:512'
}

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 7 9700X 8-Core Processor
CPU family: 26
Model: 68
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 0
Frequency boost: enabled
CPU(s) scaling MHz: 60%
CPU max MHz: 5582.0000
CPU min MHz: 600.0000
BogoMIPS: 7585.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_
good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor sss
e3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_le
gacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext
perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmc
all fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap av
x512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_
occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru w
bnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilt
er pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi
2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid bus_lock_detect movdiri movdir
64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d amd_lbr_pmc_freeze
Virtualization: AMD-V
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 8 MiB (8 instances)
L3 cache: 32 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Ghostwrite: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; IBPB on VMEXIT only
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanit
ization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIB
P always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Mitigation; IBPB on VMEXIT

Versions of relevant libraries:
[pip3] intel-openmp==2021.4.0
[pip3] mkl==2021.1.1
[pip3] mkl-devel==2021.1.1
[pip3] mkl-include==2021.1.1
[pip3] mypy_extensions==1.1.0
[pip3] numpy==1.26.4
[pip3] nv-one-logger-pytorch-lightning-integration==2.3.1
[pip3] nvidia-cuda-nvrtc==13.1.80
[pip3] nvidia-cuda-runtime==13.0.88
[pip3] nvidia-cudnn-frontend==1.16.0
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-pytriton==0.7.0
[pip3] nvtx==0.2.14
[pip3] onnx==1.18.0
[pip3] onnx_graphsurgeon==0.5.8
[pip3] onnx-ir==0.1.12
[pip3] onnxscript==0.5.7.dev20251209
[pip3] open_clip_torch==3.2.0
[pip3] optree==0.17.0
[pip3] pytorch-lightning==2.6.0
[pip3] tbb==2021.13.1
[pip3] torch==2.9.0a0+50eac811a6.nv25.9
[pip3] torch_tensorrt==2.9.0a0
[pip3] torchao==0.13.0+git
[pip3] torchdiffeq==0.2.5
[pip3] torchmetrics==1.8.2
[pip3] torchprofile==0.0.4
[pip3] torchsde==0.2.6
[pip3] torchvision==0.24.0a0+98f8b375
[pip3] torchx==0.7.0
[pip3] triton==3.4.0+gitc817b9b6
[pip3] tritonclient==2.51.0
[conda] Could not collect
```

cc @ezyang @gchanan @kadeng @msaroufim @ptrblck @eqy @jerryzh168 @tinglvv @nWEIdia