NVIDIA/TransformerEngine

View on GitHub

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Stars: 3.5k
Forks: 831
Open beginner issues: 0
Indexed issues: 163
Avg merge: 3d 11h
Merged PRs (30d): 65

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

163 open issues indexed

Preserve NaN through half-precision MXFP8 amax reductions Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#3550 · Opened Sep 19, 2026
FA4 returns an all-zero output for causal attention on SM100 after flash-attention #2490 Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

NVIDIA/TransformerEngine#3528 · 3 comments · Opened Sep 16, 2026
Graph-Safe FP8 BlockScaling Support on Blackwell for GroupedLinear Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

NVIDIA/TransformerEngine#3520 · Opened Sep 15, 2026
MXFP8 GEMM rejects compact unswizzled scales that quantize_ accepts Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

NVIDIA/TransformerEngine#3518 · 2 comments · Opened Sep 15, 2026
[PyTorch][Attention] THD P2P context-parallel regression when padded cu_seqlens are value-equal but not object-identical Open

attention

Difficulty 4/5 3-5 days Newbie friendliness 38/100

NVIDIA/TransformerEngine#3487 · Opened Sep 5, 2026
[Bug] Backend selection picks FA3 for training with head_dim_qk=192 / v_head_dim=128, but FA3 backward cannot run it Open

attention

Difficulty 2/5 1-3 hours Newbie friendliness 85/100

NVIDIA/TransformerEngine#3481 · 4 comments · Opened Sep 4, 2026
[Bug] Illegal memory access in grouped MXFP8 quantize on GB200 (sm_100) introduced by #3337; TMA store completion handoff missing in group_quantize_mxfp8.cuh Open

NVIDIA/TransformerEngine#3474 · 6 comments · 1 assignee · Opened Sep 3, 2026
Tied weights do not accumulate with delayed wgrad Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#3437 · 2 comments · Opened Aug 28, 2026
NCCL EP hard link makes `import transformer_engine` require `libcuda.so.1` Open

NVIDIA/TransformerEngine#3381 · 3 comments · 1 assignee · Opened Aug 13, 2026
[Bug] Triton mask-map moe_permute ignores num_out_tokens Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

NVIDIA/TransformerEngine#3346 · 2 comments · Opened Aug 11, 2026
[Bug] Deferred init issue in `te.pytorch.ops` Open

bug

NVIDIA/TransformerEngine#3322 · 1 comment · 1 assignee · Opened Aug 6, 2026
[Bug] CUDA graph backward replay illegal memory access after validation with mixed NVFP4/MXFP8 Mamba Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TransformerEngine#3316 · Opened Aug 4, 2026
thd + attention_dropout routes to the composite cuDNN SDPA engine and costs 5x on Blackwell Open

attention

NVIDIA/TransformerEngine#3312 · 1 assignee · Opened Aug 4, 2026
[Build] Publish PyTorch 2.11 CUDA 13 aarch64 wheels for GB200 Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#3310 · Opened Aug 4, 2026
[Build] Restore CUDA 13 Linux aarch64 core wheels Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#3309 · Opened Aug 4, 2026
[Bug] Transformer Engine SM120 (Blackwell) Compatibility Issue Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 55/100

NVIDIA/TransformerEngine#3299 · 1 comment · Opened Aug 3, 2026
[Bug] Using `QuantizedTensor` on CPU causes a `RecursionError` Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 68/100

NVIDIA/TransformerEngine#3294 · 4 comments · Opened Jul 30, 2026
[Bug] FusedAttention THD + learnable softmax backward IMA (cuDNN err 700) on Hopper Open

attention bug

NVIDIA/TransformerEngine#3249 · 6 comments · 1 assignee · Opened Jul 24, 2026
FP8 DPA future token leakage Open

attention bug

Difficulty 5/5 Over a week Newbie friendliness 38/100

NVIDIA/TransformerEngine#3248 · 1 comment · Opened Jul 24, 2026
cuDNN to accept actual seqlens in addition to cumulative seqlens Open

attention

NVIDIA/TransformerEngine#3218 · 1 comment · 1 assignee · Opened Jul 16, 2026
[BUG] NVFP4 fails on RTX 5070 Laptop (SM120): CUDA Error: no kernel image is available for execution on the device Open

Difficulty 4/5 3-5 days Newbie friendliness 50/100

NVIDIA/TransformerEngine#3217 · 2 comments · Opened Jul 16, 2026
[PyTorch] FusedAdam/multi_tensor_apply: OOB metadata writes when a zero-numel tensor lands in the last TensorListMetadata slot (CUDA IMA / misaligned address; silent corruption below threshold) Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

NVIDIA/TransformerEngine#3202 · Opened Jul 10, 2026
[PyTorch] Request a migration path for downstream `_GroupedLinear` integrations after the 2.17 signature change Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TransformerEngine#3191 · 2 comments · Opened Jul 8, 2026
Plumb dbias request in isolation into the common backend selector Open

attention bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#3188 · Opened Jul 8, 2026
Improve readability of dgrad_reduce_scatter_overlap variable name Open

NVIDIA/TransformerEngine#3164 · 1 comment · 1 assignee · Opened Jul 2, 2026
Hybrid quantization follow-ups Open

bug

NVIDIA/TransformerEngine#3158 · 1 assignee · Opened Jun 30, 2026
Track recipe config changes for quantizer rebuilds Open

bug

NVIDIA/TransformerEngine#3157 · 1 assignee · Opened Jun 30, 2026
Multi-GPU FP8 (accelerate.dispatch_model + DelayedScaling HYBRID) — illegal memory access on H200 sm_90 Open

Difficulty 5/5 Over a week Newbie friendliness 38/100

NVIDIA/TransformerEngine#3124 · 3 comments · Opened Jun 12, 2026
Vendor-neutral reference: NVFP4 vs OCP MXFP4 block-structure parameters (request for confirmation) Open

question

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#3105 · Opened Jun 8, 2026
`test_comm_gemm_overlap.py::test_multi_layer_with_overlap_bf16` fails on A100 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#3097 · Opened Jun 5, 2026
Low-overhead stateless functional quantize/GEMM API (plain tensors in → out) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 32/100

NVIDIA/TransformerEngine#3087 · 1 comment · Opened Jun 5, 2026
Fused attention should distinguish bias input from dBias request Open

attention

NVIDIA/TransformerEngine#3082 · 1 assignee · Opened Jun 4, 2026
NVFP4 default recipe (RHT + stochastic rounding) crashes on sm_120: fused kernel exceeds 101376-byte shared-memory opt-in cap Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#3062 · 1 reaction · Opened May 29, 2026
[Common] Move heavy kernel compilation to NVRTC Open

installation

Difficulty 5/5 Over a week Newbie friendliness 45/100

NVIDIA/TransformerEngine#3054 · 1 comment · Opened May 28, 2026
[JAX] score_mod attention to support "enforce_precompiled" flag in cuDNN graph deserialization Open

attention jax

NVIDIA/TransformerEngine#3046 · 1 assignee · Opened May 26, 2026
[PyTorch] Integrate cuDNN GQA + DSA backend into DotProductAttention Open

attention

NVIDIA/TransformerEngine#3028 · 1 reaction · 1 assignee · Opened May 21, 2026
[Common + PyTorch] Support Dispatch FP8 Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#3026 · Opened May 21, 2026
[JAX] Dispatch/Combine for BF16 Tensor Open

NVIDIA/TransformerEngine#3025 · 1 comment · 1 assignee · Opened May 21, 2026
[PyTorch] Dispatch/Combine for BF16 tensor + Zero-Copy Open

NVIDIA/TransformerEngine#3024 · 1 comment · 1 assignee · Opened May 21, 2026
[Common] Initial NCCL EP integration + Distributed CPP unit tests Open

NVIDIA/TransformerEngine#3023 · 1 comment · 1 assignee · Opened May 21, 2026
[Common] Infrastructure for symmetric window Open

NVIDIA/TransformerEngine#3022 · 1 assignee · Opened May 21, 2026
TE EP comm backend with NCCL EP Open

NVIDIA/TransformerEngine#3021 · 1 assignee · Opened May 21, 2026
No flash and fused attention for Gemma4 26B-A4B Open

bug megatron

Difficulty 4/5 3-5 days Newbie friendliness 42/100

NVIDIA/TransformerEngine#3006 · 1 comment · Opened May 18, 2026
[JAX] Move quantization to before AG in MoEBlock Open

NVIDIA/TransformerEngine#2998 · 1 assignee · Opened May 15, 2026
Reduce CPU overheads in te Sequential GroupedLinear Op Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

NVIDIA/TransformerEngine#2995 · 2 comments · Opened May 14, 2026
Activation + Group Quantize Fusion with te.Sequential Open

enhancement

NVIDIA/TransformerEngine#2988 · 2 comments · 1 assignee · Opened May 13, 2026
Expose Batch Invariant Kernels Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

NVIDIA/TransformerEngine#2986 · 1 comment · Opened May 13, 2026
TransformerLayer implementation is overly complex and difficult to follow Open

question

Difficulty 5/5 Over a week Newbie friendliness 30/100

NVIDIA/TransformerEngine#2903 · Opened Apr 20, 2026
ub_overlap_ag_fprop disabled during activation recomputation due to is_grad_enabled check Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 55/100

NVIDIA/TransformerEngine#2902 · 3 comments · Opened Apr 20, 2026
[BUG] `CrossEntropyFunction.forward()` modifies input in-place without `ctx.mark_dirty()`, causing GPU memory leak Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 52/100

NVIDIA/TransformerEngine#2899 · 2 comments · Opened Apr 17, 2026

NVIDIA/TransformerEngine

Get new issues in your inbox