NVIDIA / NVIDIA/Megatron-LM

Do not auto select NVTE FUSED if CUDNN_FRONTEND version is less than 1.15.0

Open
#2,011 0 comments 0 reactions 0 assignees View on GitHub
bug community-request module: transformer engine
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 3h
Merged PRs (30d)
272

Description

**Describe the bug**

A clear and concise description of what the bug is.

The heuristic for selecting NVTE-FUSED is aggressive and can accidentally trigger non-deterministic NaNs on the SDPA attention when dispatched via CUDNN frontend (fixed in most recently release). Should guarand against all but the newer transformer engine releases. Latest PyTorch bug discussion is here: https://github.com/pytorch/pytorch/issues/166211

**Steps/Code to reproduce bug**

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.

A helpful guide on on how to craft a minimal bug report http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Additional context**

Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by locating the heuristic that selects NVTE-FUSED and how it detects the CUDNN frontend version; confirm the relevant Transformer Engine or PyTorch integration path. Done means NVTE-FUSED is not automatically selected when CUDNN_FRONTEND is older than 1.15.0, while newer supported versions retain the intended behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.