NVIDIA / NVIDIA/TransformerEngine

RuntimeError with Assertion failed: driver_result == cudaDriverEntryPointSuccess.

Open
#2,258 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
3.5k
Forks
831
Avg merge
3d 11h
Merged PRs (30d)
65

Description

Describe the bug

when running examples/llama/train_llama3_8b_fp8.sh and building transformer layer of GPTModel:

RuntimeError: /TransformerEngine/transformer_engine/common/util/cuda_driver.cpp:42 in function get_symbol: Assertion failed: driver_result == cudaDriverEntryPointSuccess. Could not find CUDA driver entry point for cuCtxGetCurrent when instantiating TERowParallelLinear when instantiating SelfAttentio

Steps/Code to reproduce bug

  1. conda create -n megatronlm python=3.12
  2. pip install torch==2.6.0
  3. pip install megatron-core
  4. pip install --no-build-isolation transformer-engine[pytorch]
  5. Pip install regex six PyYAML psutil pybind11

then just run examples/llama/train_llama3_8b_fp8.sh

Environment overview (please complete the following information)
ubuntu 22.04
python 3.12
torch 2.6.0+cu124
cuda 12.4(by nvcc)
cudnn 9.1.0
tranformer-engine 2.8.0

Device details
H100

Additional context

Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/llama/train_llama3_8b_fp8.sh and inspect transformer_engine/common/util/cuda_driver.cpp at line 42, especially get_symbol and the cuCtxGetCurrent lookup. Reproduce the failure with the listed PyTorch, CUDA, H100, and Transformer Engine versions, then trace the TERowParallelLinear and SelfAttention instantiation path. Done means the example proceeds without this assertion under the reported environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch, shell
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.