NVIDIA / NVIDIA/TransformerEngine
RuntimeError with Assertion failed: driver_result == cudaDriverEntryPointSuccess.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.5k
- Forks
- 831
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 65
Description
Describe the bug
when running examples/llama/train_llama3_8b_fp8.sh and building transformer layer of GPTModel:
RuntimeError: /TransformerEngine/transformer_engine/common/util/cuda_driver.cpp:42 in function get_symbol: Assertion failed: driver_result == cudaDriverEntryPointSuccess. Could not find CUDA driver entry point for cuCtxGetCurrent when instantiating TERowParallelLinear when instantiating SelfAttentio
Steps/Code to reproduce bug
- conda create -n megatronlm python=3.12
- pip install torch==2.6.0
- pip install megatron-core
- pip install --no-build-isolation transformer-engine[pytorch]
- Pip install regex six PyYAML psutil pybind11
then just run examples/llama/train_llama3_8b_fp8.sh
Environment overview (please complete the following information)
ubuntu 22.04
python 3.12
torch 2.6.0+cu124
cuda 12.4(by nvcc)
cudnn 9.1.0
tranformer-engine 2.8.0
Device details
H100
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/llama/train_llama3_8b_fp8.sh and inspect transformer_engine/common/util/cuda_driver.cpp at line 42, especially get_symbol and the cuCtxGetCurrent lookup. Reproduce the failure with the listed PyTorch, CUDA, H100, and Transformer Engine versions, then trace the TERowParallelLinear and SelfAttention instantiation path. Done means the example proceeds without this assertion under the reported environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch, shell
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100