pytorch / pytorch/pytorch

Getting fallback when using scaled_dot_product_attention() despite CUDA kernels being installed and available

Open
#167,754 0 comments 0 reactions 0 assignees View on GitHub
module: sdpa triaged
Dominant language
Python
Stars
103k
Forks
29.6k
PR merge metrics
PR metrics pending

Description

### 🐛 Describe the bug

Bug:
```
UserWarning: Memory efficient kernel not used because: (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/transformers/cuda/sdp_utils.cpp:906.)
out = F.scaled_dot_product_attention(q, k, v, dropout_p=dropout_p)
UserWarning: Memory Efficient attention has been runtime disabled. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/transformers/sdp_utils_cpp.h:552.)
out = F.scaled_dot_product_attention(q, k, v, dropout_p=dropout_p)
UserWarning: Flash attention kernel not used because: (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/transformers/cuda/sdp_utils.cpp:908.)
out = F.scaled_dot_product_attention(q, k, v, dropout_p=dropout_p)
UserWarning: Expected query, key and value to all be of dtype: {Half, BFloat16}. Got Query dtype: float, Key dtype: float, and Value dtype: float instead. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/transformers/sdp_utils_cpp.h:91.)
out = F.scaled_dot_product_attention(q, k, v, dropout_p=dropout_p)
UserWarning: cuDNN attention kernel not used because: (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/transformers/cuda/sdp_utils.cpp:910.)
out = F.scaled_dot_product_attention(q, k, v, dropout_p=dropout_p)
/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py:1786: UserWarning: Flash Attention kernel failed due to: No available kernel. Aborting execution.
Falling back to all available kernels for scaled_dot_product_attention (which may have a slower speed).

```
This is running on an ARM device (Jetson Orin). I tried using a pytorch container from NVIDIA's supported [images](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags?version=25.10-py3-igpu). I also tried downloading kernels manually from the NVIDIA [index](https://pypi.jetson-ai-lab.io/jp6/cu126). **Seems like pytorch isn't able to use the installed CUDA kernels.**

### Versions

PyTorch version: 2.9.0a0+145a3a7bda.nv25.10
Is debug build: False
CUDA used to build PyTorch: 12.9
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.3 LTS (aarch64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.39

Python version: 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.15.148-tegra-aarch64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: 12.9.86
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Orin (nvgpu)
Nvidia driver version: 540.4.0
cuDNN version: Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.9.10.2
/usr/lib/aarch64-linux-gnu/libcudnn_adv.so.9.10.2
/usr/lib/aarch64-linux-gnu/libcudnn_cnn.so.9.10.2
/usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.10.2
/usr/lib/aarch64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.10.2
/usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.10.2
/usr/lib/aarch64-linux-gnu/libcudnn_heuristic.so.9.10.2
/usr/lib/aarch64-linux-gnu/libcudnn_ops.so.9.10.2
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: ARM
Model name: Cortex-A78AE
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 4
Socket(s): -
Cluster(s): 2
Stepping: r0p1
CPU(s) scaling MHz: 58%
CPU max MHz: 1984.0000
CPU min MHz: 115.2000
BogoMIPS: 62.50
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp uscat ilrcpc flagm paca pacg
L1d cache: 512 KiB (8 instances)
L1i cache: 512 KiB (8 instances)
L2 cache: 2 MiB (8 instances)
L3 cache: 4 MiB (2 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, but not BHB
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.1.0
[pip3] nvidia-cudnn-frontend==1.12.0
[pip3] onnx==1.18.0
[pip3] optree==0.17.0
[pip3] pytorch-triton==3.4.0+gitc817b9b6
[pip3] torch==2.9.0a0+145a3a7bda.nv25.10
[pip3] torch_tensorrt==2.9.0a0
[pip3] torchprofile==0.0.4
[pip3] torchvision==0.24.0a0+094e7af5
[conda] Could not collect

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.