NVIDIA/TransformerEngine

View on GitHub

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Stars: 3.5k
Forks: 831
Open beginner issues: 0
Indexed issues: 163
Avg merge: 3d 11h
Merged PRs (30d): 65

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

163 open issues indexed

[PyTorch] Reduce CPU overhead in grouped MLP block Open

cpu_overhead performance

NVIDIA/TransformerEngine#2897 · 2 comments · 1 reaction · 3 assignees · Opened Apr 16, 2026
[BUG] SIGSEGV in fused_attn_fwd when cu_seqlens_q != cu_seqlens_q_padded with FP8 blockwise Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

NVIDIA/TransformerEngine#2892 · 2 comments · Opened Apr 16, 2026
[Bug]trtexec build engine succeeded on the H100 GPU but failed on the 5090 GPU Open

bug

NVIDIA/TransformerEngine#2891 · 4 comments · 1 assignee · Opened Apr 16, 2026
[Feature Request] Add Gated Delta Net (GDN) support Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2884 · Opened Apr 15, 2026
[Bug] Context parallel crashes with asymmetric K/V head dims (GQA + enable_mla path) Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 68/100

NVIDIA/TransformerEngine#2868 · 3 comments · Opened Apr 11, 2026
FP8 weight caching shows no speedup (sometimes slowdown) and causes training curve differences under Float8BlockScaling Open

NVIDIA/TransformerEngine#2852 · 2 comments · 1 assignee · Opened Apr 8, 2026
Request for batched general_gemm() (or FP8-aware torch.bmm) for non-Linear GEMM workloads Open

field-request

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TransformerEngine#2846 · Opened Apr 7, 2026
Questions about test_float8_blockwise_gemm_exact.py Open

question

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#2828 · Opened Apr 3, 2026
Adaptive Compression Open

question

Difficulty 5/5 Over a week Newbie friendliness 30/100

NVIDIA/TransformerEngine#2807 · 4 comments · 2 reactions · Opened Mar 28, 2026
Request for additional features from TE Debug modules for low precision training. Open

enhancement

NVIDIA/TransformerEngine#2801 · 2 comments · 1 assignee · Opened Mar 25, 2026
[Question] Expected behavior for blockwise FP8? Hybrid E4M3/E5M2 format & eval metrics outperforming BF16 Open

question

NVIDIA/TransformerEngine#2754 · 2 comments · 1 assignee · Opened Mar 11, 2026
Dequantization support for the grouped tensor - NVFP4 Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#2726 · Opened Mar 2, 2026
Enable Router custom call from JAX Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#2710 · Opened Feb 26, 2026
Performance bottleneck in TransformerLayer and Attention when using attention mask Open

bug

NVIDIA/TransformerEngine#2703 · 1 comment · 1 assignee · Opened Feb 25, 2026
[JAX] Expose sigmoid GLU activation type Open

jax Priority = P1

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#2687 · 2 comments · Opened Feb 17, 2026
[Inquiry] Support status and roadmap for MXFP8 on SM120 (Blackwell) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2668 · 3 comments · 3 reactions · Opened Feb 10, 2026
[Core] TE common nvte_grouped_gemm workspace is incorrect Open

bug MoE

Difficulty 5/5 Over a week Newbie friendliness 5/100

NVIDIA/TransformerEngine#2660 · Opened Feb 9, 2026
[JAX] Upstream checkpointing of quantizations in TE/JAX to MaxText Open

field-request fp4 FP8 jax

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#2658 · 1 comment · Opened Feb 6, 2026
Improve comm_gemm_overlap documentation and HuggingFace integration guidance Open

documentation

Difficulty 3/5 1-2 days Newbie friendliness 52/100

NVIDIA/TransformerEngine#2618 · Opened Jan 23, 2026
Heuristic Fallback in cuDNN: Why THD Format Reverts to SM80 on Hopper GPUs Open

question

NVIDIA/TransformerEngine#2616 · 2 comments · 1 assignee · Opened Jan 22, 2026
[PyTorch] Torch.compile support Open

enhancement

NVIDIA/TransformerEngine#2590 · 12 comments · 2 reactions · 1 assignee · Opened Jan 13, 2026
Enable post-RHT amax estimation Open

enhancement fp4

NVIDIA/TransformerEngine#2578 · 1 assignee · Opened Jan 8, 2026
How to use jax.triton_extensions without having to build the project or copy the source code Open

build jax

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2571 · 2 comments · Opened Jan 7, 2026
NVFP4 Selective Fwd / Bwd Open

fp4 question

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TransformerEngine#2569 · 1 comment · Opened Jan 7, 2026
Why NVFP4 Inference (50 PFLOPS) Outperforms Training (35 PFLOPS) on Rubin GPU？ Open

fp4 question

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2565 · 4 comments · Opened Jan 6, 2026
Support on Thor? Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 15/100

NVIDIA/TransformerEngine#2556 · Opened Jan 4, 2026
[MX] Rowwise `W` cached for backwards Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#2546 · 2 comments · 1 reaction · Opened Dec 29, 2025
[JAX] Support vmapped cudnn attention Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

NVIDIA/TransformerEngine#2545 · 1 comment · 1 reaction · Opened Dec 26, 2025
using cuda_graph memory increased a lot and oom at torch.autograd.grad Open

bug megatron

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#2543 · Opened Dec 25, 2025
no attention backend support for softmax = learnable and cp_comm_type = p2p Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 42/100

NVIDIA/TransformerEngine#2541 · 2 comments · 1 reaction · Opened Dec 23, 2025
Linear layer reset_parameters() changes bias zero init to random init near 0 Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

NVIDIA/TransformerEngine#2529 · 2 comments · 1 reaction · Opened Dec 17, 2025
What is the updated NVFP4 recipe for Nemotron 3 SUPER/ULTRA? Open

fp4 question

Difficulty 4/5 3-5 days Newbie friendliness 25/100

NVIDIA/TransformerEngine#2526 · 1 reaction · Opened Dec 17, 2025
[Docs] Add datatype × hardware support matrix Open

documentation

Difficulty 4/5 3-5 days Newbie friendliness 45/100

NVIDIA/TransformerEngine#2512 · Opened Dec 11, 2025
[Feature Request] Add Native Sparse Attention (NSA) support Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TransformerEngine#2511 · 1 comment · 2 reactions · Opened Dec 11, 2025
Examples: Using TE with LLaMA-Factory Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 38/100

NVIDIA/TransformerEngine#2509 · Opened Dec 11, 2025
Problem encountered while compiling version v2.6 Open

bug installation

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#2504 · 7 comments · Opened Dec 11, 2025
[PyTorch] Reduce CPU overhead in attention Open

attention

NVIDIA/TransformerEngine#2498 · 2 assignees · Opened Dec 9, 2025
[Jax] Non-CP attention documentation Open

attention

NVIDIA/TransformerEngine#2497 · 1 assignee · Opened Dec 9, 2025
[PyTorch/Jax] CP attention documentation Open

attention

NVIDIA/TransformerEngine#2496 · 2 assignees · Opened Dec 9, 2025
[Jax] CP + striping + AG vs. CP + striping + P2P performance comparison Open

attention

NVIDIA/TransformerEngine#2495 · 1 assignee · Opened Dec 9, 2025
softmax_type != vanilla support thd format Open

attention

NVIDIA/TransformerEngine#2483 · 5 comments · 1 assignee · Opened Dec 5, 2025
[JAX] Integrate grouped tensor support into MaxText Open

2.17 MoE

Difficulty 5/5 Over a week Newbie friendliness 30/100

NVIDIA/TransformerEngine#2472 · 2 comments · Opened Dec 3, 2025
[JAX] Remove D2H in GroupedQuantizeFFI/GroupedGemmFFI Open

2.17 MoE

NVIDIA/TransformerEngine#2469 · 1 comment · 1 assignee · Opened Dec 3, 2025
Activation support for GroupedTensor: BF16→BF16 version Open

MoE

NVIDIA/TransformerEngine#2458 · 1 comment · 1 assignee · Opened Dec 3, 2025
Inconsistent fused_attn_bwd results of dk, dv under THD layout + GQA on multiple calls Open

attention bug

NVIDIA/TransformerEngine#2448 · 2 comments · 1 assignee · Opened Dec 3, 2025
MoE training optimization Open

MoE

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/TransformerEngine#2438 · 1 reaction · Opened Dec 1, 2025
Question on https://arxiv.org/pdf/2509.25149 Open

Difficulty 3/5 1-2 days Newbie friendliness 15/100

NVIDIA/TransformerEngine#2433 · Opened Nov 28, 2025
Compatibility issues between FusedAdam and DTensor Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 45/100

NVIDIA/TransformerEngine#2424 · Opened Nov 26, 2025
nan in tests/jax/test_distributed_layernorm_mlp.py::TestDistributedLayernormMLP::test_layernorm_mlp_grad Open

bug FP8 jax waiting-for-feedback

Difficulty 4/5 3-5 days Newbie friendliness 45/100

NVIDIA/TransformerEngine#2415 · 3 comments · Opened Nov 24, 2025
AssertionError: Could not find `transformer-engine` PyPI package Open

bug installation

Difficulty 3/5 1-2 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#2410 · 7 comments · Opened Nov 21, 2025

NVIDIA/TransformerEngine

Get new issues in your inbox