NVIDIA/TransformerEngine

View on GitHub

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Stars: 3.5k
Forks: 831
Open beginner issues: 0
Indexed issues: 163
Avg merge: 3d 11h
Merged PRs (30d): 65

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

163 open issues indexed

How to use thd format qkv with cp + packed_seq_params Open

question

Difficulty 5/5 Over a week Newbie friendliness 15/100

NVIDIA/TransformerEngine#1368 · 4 comments · Opened Dec 12, 2024
Support more than 1 shape/attention_params for DotProductAttention decision cache Open

enhancement

NVIDIA/TransformerEngine#1349 · 1 assignee · Opened Nov 29, 2024
[QUESTION] Does TP overlap support variable sequence length? Open

question

Difficulty 5/5 Over a week Newbie friendliness 20/100

NVIDIA/TransformerEngine#1303 · 7 comments · Opened Nov 1, 2024
FSDP with FP8 is not working Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 45/100

NVIDIA/TransformerEngine#1247 · 2 comments · Opened Oct 14, 2024
RMSNorm precision different from HF implementation Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

NVIDIA/TransformerEngine#1132 · 5 comments · Opened Aug 23, 2024
Calling backward(retain_graph=True) multiple times with TE Layer does not work Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

NVIDIA/TransformerEngine#990 · 3 comments · Opened Jul 4, 2024
AttnFuncWithCP can use less memory Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#952 · 1 reaction · Opened Jun 21, 2024
Why use two streams for context parallel Open

question

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#947 · 3 comments · Opened Jun 19, 2024
'TEDotProductAttention' object has no attribute 'tp_group_initialized' Open

bug megatron

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#934 · 3 comments · Opened Jun 17, 2024
[Feature Request][PyTorch] Support thd format for fp8 tensors in DotProductAttention Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TransformerEngine#923 · 1 comment · 1 reaction · Opened Jun 13, 2024
[Feature Request] Integration of DiT components into TransformerEngine. Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

NVIDIA/TransformerEngine#900 · 1 comment · 2 reactions · Opened Jun 10, 2024
Request for Adaptive Layer Norm MLP Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 38/100

NVIDIA/TransformerEngine#789 · 9 comments · Opened Apr 17, 2024
Add: transformerengine.test() Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 48/100

NVIDIA/TransformerEngine#170 · 1 comment · Opened Apr 24, 2023

NVIDIA/TransformerEngine

Get new issues in your inbox