NVIDIA/TransformerEngine

View on GitHub

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Stars: 3.5k
Forks: 831
Open beginner issues: 0
Indexed issues: 163
Avg merge: 3d 11h
Merged PRs (30d): 65

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

163 open issues indexed

Grouped Linear issue Open

bug needinfo

Difficulty 4/5 3-5 days Newbie friendliness 42/100

NVIDIA/TransformerEngine#2409 · 1 comment · Opened Nov 21, 2025
CUDA extension loading should be modular Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#2405 · 1 comment · 1 reaction · Opened Nov 20, 2025
MXFP4 support Open

enhancement fp4

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/TransformerEngine#2404 · Opened Nov 20, 2025
FP8BlockQuantizer not work on TE Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#2393 · 10 comments · Opened Nov 17, 2025
Release training examples for https://www.arxiv.org/pdf/2509.25149 Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2389 · 1 comment · 1 reaction · Opened Nov 14, 2025
Error when compiling from source with clang++ as nvcc host compiler Open

build

Difficulty 4/5 3-5 days Newbie friendliness 30/100

NVIDIA/TransformerEngine#2384 · 1 reaction · Opened Nov 14, 2025
[BUG] fp8blockscaled training not converging on sm120 Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2382 · 1 comment · Opened Nov 14, 2025
Hadamard transform not working on SM120 Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#2372 · 1 comment · 1 reaction · Opened Nov 12, 2025
Restructure attention C API Open

2.16.0 attention refactor

NVIDIA/TransformerEngine#2335 · 2 assignees · Opened Nov 2, 2025
MuonClip for Kimi-K2 model Open

attention

NVIDIA/TransformerEngine#2334 · 2 comments · 2 assignees · Opened Nov 2, 2025
Distributed AdamW Optimizer for DDP training? Open

question

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2327 · Opened Oct 31, 2025
Question regarding the 2D weight quantization and quality degradation in 1D case Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/TransformerEngine#2314 · 2 comments · Opened Oct 29, 2025
Document Supported Datatypes Per Compute Capability Open

documentation

Difficulty 4/5 3-5 days Newbie friendliness 38/100

NVIDIA/TransformerEngine#2304 · 7 reactions · Opened Oct 26, 2025
RuntimeError with Assertion failed: driver_result == cudaDriverEntryPointSuccess. Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#2258 · 5 comments · Opened Oct 10, 2025
is NVFP4 not supported for rtx 50 series? Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

NVIDIA/TransformerEngine#2255 · 26 comments · 5 reactions · Opened Oct 9, 2025
Pre-built wheels Open

build installation

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2249 · 5 comments · 1 reaction · Opened Oct 8, 2025
Discuss release of NVFP4-trained LLM on Hugging Face Open

megatron

NVIDIA/TransformerEngine#2224 · 3 comments · 2 reactions · 1 assignee · Opened Oct 1, 2025
thd format is not supported with hierarchical CP implementation yet Open

attention

NVIDIA/TransformerEngine#2208 · 6 comments · 1 assignee · Opened Sep 28, 2025
Potential Performance Regression in v2.9.0 due to CUTLASS kernels GroupedGemm on large token sizes Open

bug

NVIDIA/TransformerEngine#2203 · 3 comments · 1 assignee · Opened Sep 25, 2025
Increase MAX_TENSOR_NUM Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

NVIDIA/TransformerEngine#2189 · 7 comments · 5 reactions · Opened Sep 19, 2025
Using system CUDA libraries Open

bug installation

Difficulty 3/5 1-2 days Newbie friendliness 56/100

NVIDIA/TransformerEngine#2147 · 3 comments · 2 reactions · Opened Sep 2, 2025
Performance Issue with NVIDIA Transformer Engine FP8 Linear Functions on L20 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#2126 · 10 comments · Opened Aug 27, 2025
[PyTorch] High CPU overhead on Grace systems Open

bug

Difficulty 5/5 Over a week Newbie friendliness 32/100

NVIDIA/TransformerEngine#2053 · 8 comments · Opened Aug 11, 2025
2D by 2D per-block quantization in FP8 GEMM Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2025 · 3 comments · Opened Aug 4, 2025
_flash_attn_forward() missing `v_descale` when enabling context parallel Open

attention bug

NVIDIA/TransformerEngine#1987 · 1 comment · 1 assignee · Opened Jul 23, 2025
Memory leak in fused attention + fp8 Open

attention bug

NVIDIA/TransformerEngine#1935 · 1 comment · 1 assignee · Opened Jul 8, 2025
Does Transformer Engine support sequence lengths that are not divisible by CP? Open

attention

NVIDIA/TransformerEngine#1931 · 1 comment · 1 assignee · Opened Jul 7, 2025
There is a bug when both packing and context parallelism (CP) are enabled at the same time. Open

attention bug

NVIDIA/TransformerEngine#1930 · 3 comments · 1 assignee · Opened Jul 6, 2025
[bug] attention benchmark script not work Open

attention

NVIDIA/TransformerEngine#1919 · 1 comment · 1 assignee · Opened Jul 3, 2025
comm_gemm_overlap with atomic gemm + fp8 hangs Open

question

Difficulty 4/5 3-5 days Newbie friendliness 32/100

NVIDIA/TransformerEngine#1918 · Opened Jul 2, 2025
How to Perform Full Cross-Attention with CP Enabled and Padding Mask Open

attention

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#1915 · 1 comment · Opened Jul 1, 2025
whl contains .cpp files Open

build question

Difficulty 3/5 1-2 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#1893 · 2 comments · Opened Jun 25, 2025
Storage in fp8 Open

question

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#1880 · 8 comments · Opened Jun 13, 2025
DCP Checkpoint Load Fails for _extra_state when training in FSDP2 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#1860 · 3 comments · Opened Jun 9, 2025
can't install te v1.13 in databricks Open

build installation

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#1782 · 3 comments · Opened May 14, 2025
Create option to control data type for tensor parallel all-reduce Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 52/100

NVIDIA/TransformerEngine#1761 · 4 comments · Opened May 8, 2025
something about atomic gemm Open

question

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#1760 · Opened May 8, 2025
B200 Slow GEMM (16384, 1024, 8192) Only when run in a particular order Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#1747 · 1 comment · Opened May 3, 2025
[BUG] Inconsistent LayerNorm Parameter Gradient with TP+CP+FP8 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 38/100

NVIDIA/TransformerEngine#1677 · 1 comment · Opened Apr 12, 2025
[BUG] Wrong attention gradient in Transformer Engine Open

attention bug

NVIDIA/TransformerEngine#1615 · 4 comments · 1 reaction · 1 assignee · Opened Mar 26, 2025
did not get improvement from tp/sp overlap Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#1587 · 1 comment · Opened Mar 18, 2025
curious about atomic_gemm_overlap_ag Open

question

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#1583 · Opened Mar 17, 2025
How can we use te.Linear with weight parallel? Open

question

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#1532 · 3 comments · Opened Mar 4, 2025
Question about the performace of GroupedLinear Open

performance

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/TransformerEngine#1499 · 7 comments · Opened Feb 20, 2025
`Float8Quantizer::create_tensor` calculates `scale_inv` instead of creating an empty buffer Open

performance

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#1491 · 3 comments · Opened Feb 18, 2025
Questions about accuracy alignment between BF16 and FP8 Open

question

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#1419 · 2 comments · Opened Jan 22, 2025
Fused attention error while running Nvidia Cosmos Open

attention

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#1407 · 4 comments · Opened Jan 13, 2025
why close ag overlap when is_grad_enabled is False Open

question

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#1398 · 1 comment · 2 reactions · Opened Jan 10, 2025
FP8 GEMM Kernels Open

question

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/TransformerEngine#1391 · 2 comments · 1 reaction · Opened Jan 6, 2025
thd qkv-format in transformer layer Open

attention

Difficulty 4/5 3-5 days Newbie friendliness 30/100

NVIDIA/TransformerEngine#1383 · Opened Dec 22, 2024

NVIDIA/TransformerEngine

Get new issues in your inbox