pytorch/torchtitan

View on GitHub

A PyTorch native platform for training generative AI models

Stars: 5.7k
Forks: 994
Open beginner issues: 2
Indexed issues: 231
Avg merge: 2d 5h
Merged PRs (30d): 238

Dominant language: Python
License: BSD-3-Clause
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

231 open issues indexed

[transformers_modeling_backend]Regression of #3775: rotary inv_freq is left uninitialized after `to_empty()` Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

pytorch/torchtitan#4783 · 3 comments · 1 reaction · Opened Sep 18, 2026
[Bug] An asynchronous TorchFT healing export reshards an in-flight FSDP2 parameter Open

pytorch/torchtitan#4753 · 1 assignee · Opened Sep 16, 2026
[RL] #4535 (fused gate-up default) makes RL weight sync copy the whole FFN projection every step Open

pytorch/torchtitan#4706 · 3 comments · 1 assignee · Opened Sep 15, 2026
[RFC][DistMuon] Handle fused weights correctly for Kimi K3 per-head Muon Open

pytorch/torchtitan#4692 · 3 comments · 2 assignees · Opened Sep 15, 2026
[graph_trainer] SAR omits in-place writes to recomputed buffers, producing incorrect gradients Open

Difficulty 4/5 3-5 days Newbie friendliness 56/100

pytorch/torchtitan#4688 · 1 comment · Opened Sep 15, 2026
[MTP] Use per-module valid-token denominators for AuxLoss Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

pytorch/torchtitan#4682 · 1 comment · Opened Sep 14, 2026
[graph_trainer][pipelining] Carry schedule metadata into reusable stage graphs Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

pytorch/torchtitan#4655 · Opened Sep 14, 2026
[Bug][DistMuon] split fused gated linear weights before NS computation Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

pytorch/torchtitan#4532 · Opened Sep 9, 2026
[Bug] Weight initialized from the same seed when cpu offloading is active Open

high priority triage review

pytorch/torchtitan#4511 · 1 comment · 1 assignee · Opened Sep 7, 2026
[Qwen3 MoE] Make auxiliary-loss-free expert-bias updates explicit and migration-safe Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

pytorch/torchtitan#4394 · 5 comments · Opened Sep 1, 2026
CUDAGraphable MoE Status Open

pytorch/torchtitan#4367 · 2 assignees · Opened Aug 28, 2026
[RFC] Support Anticipatory Routing to mitigate MoE training loss spikes Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

pytorch/torchtitan#4349 · 1 comment · Opened Aug 27, 2026
[Graph Trainer] [EP Overlap] Remove transformer-level chunking Open

pytorch/torchtitan#4342 · 1 assignee · Opened Aug 27, 2026
SSRF: arbitrary URL fetch in image.py:36 without private IP validation Open

Difficulty 3/5 1-2 days Newbie friendliness 65/100

pytorch/torchtitan#4293 · 1 reaction · Opened Aug 23, 2026
Kimi K3 issue tracking Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

pytorch/torchtitan#4272 · 11 comments · Opened Aug 21, 2026
[Kimi K3] Add MXFP4-weight / MXFP8-activation QAT and checkpoint interoperability Open

Difficulty 5/5 Over a week Newbie friendliness 28/100

pytorch/torchtitan#4269 · 3 comments · Opened Aug 21, 2026
[RFC] Make TorchFT checkpoint healing persistent and FSDP-safe Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

pytorch/torchtitan#4209 · Opened Aug 19, 2026
[RFC] Complete synchronous TorchFT integration Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

pytorch/torchtitan#4208 · Opened Aug 19, 2026
[RFC] Expose LocalSGD CPU staging in TorchTitan Open

Difficulty 2/5 Half a day Newbie friendliness 76/100

pytorch/torchtitan#4207 · Opened Aug 19, 2026
[RFC] Add an extensible TorchFT process-group registry Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

pytorch/torchtitan#4206 · Opened Aug 19, 2026
CI misconfiguration prevents some jobs running on stack PRs Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

pytorch/torchtitan#4125 · 1 comment · 1 reaction · Opened Aug 12, 2026
`cp_spmd_types` hangs on ROCm CI: CP redistribution collectives race FSDP on `default_pg` Open

module: rocm

Difficulty 4/5 3-5 days Newbie friendliness 55/100

pytorch/torchtitan#4064 · 1 comment · Opened Aug 4, 2026
GraphTrainer H100 EP-overlap MoE tests fail moving FSDP comms to dense regions Open

pytorch/torchtitan#4052 · 1 comment · 3 assignees · Opened Aug 1, 2026
GraphTrainer H100 DeepSeekV3 FSDP+TP+EP regional_inductor fails in FSDP bucketing topological sort Open

pytorch/torchtitan#4047 · 1 comment · 1 assignee · Opened Aug 1, 2026
CI runner races torchtitan checkout Open

Difficulty 2/5 1-3 hours Newbie friendliness 64/100

pytorch/torchtitan#4041 · 2 comments · Opened Jul 31, 2026
[rl] Trainer/generator GPUs sit idle from CPU thread oversubscription Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

pytorch/torchtitan#4038 · 3 comments · Opened Jul 31, 2026
[RFC] DCP Checkpoint Load: Striped I/O + Collective Broadcast Open

pytorch/torchtitan#4035 · 6 comments · 1 assignee · Opened Jul 31, 2026
[RFC] Support loose checkpoint loading via checkpoint.ignore_unexpected Open

pytorch/torchtitan#4010 · 11 comments · 1 assignee · Opened Jul 29, 2026
VLM MFU is miscomputed: vision-encoder FLOPs billed per decoder token, not per image patch Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

pytorch/torchtitan#4009 · 1 comment · Opened Jul 29, 2026
[feat] RFC: FP8 compilation support in GraphTrainer Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

pytorch/torchtitan#3990 · 5 comments · Opened Jul 27, 2026
checkpoint merge & model averaging via dummy optimizer Open

high priority triage review

Difficulty 5/5 Over a week Newbie friendliness 25/100

pytorch/torchtitan#3982 · 5 comments · 1 reaction · Opened Jul 24, 2026
Compiled FlexAttention is nondeterministic on ROCm (gfx950) Open

module: rocm

Difficulty 4/5 3-5 days Newbie friendliness 35/100

pytorch/torchtitan#3960 · Opened Jul 21, 2026
Discussion: Deprecate ModelSpec and migrate its fields Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

pytorch/torchtitan#3952 · 4 comments · Opened Jul 20, 2026
[rl] Flex-attention generator crashes with "Dynamo recompile limit exceeded (recompile_limit=8)" Open

module: torch.compile

pytorch/torchtitan#3898 · 2 comments · 1 assignee · Opened Jul 10, 2026
Add the required_grads of parameter to register_parameter Open

Difficulty 2/5 1-3 hours Newbie friendliness 25/100

pytorch/torchtitan#3885 · Opened Jul 8, 2026
[graph_trainer] Skip all test_eager_self_deterministic: hardcoded eager goldens unmaintainable vs moving torch nightly Open

Difficulty 2/5 1-3 hours Newbie friendliness 20/100

pytorch/torchtitan#3876 · Opened Jul 7, 2026
[graph_trainer] Flaky: test_graph_pp_moe_dsv3_aot_fx_trace_vs_eager (DSv3 MoE EP all-to-all nondeterminism under --debug.deterministic) Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

pytorch/torchtitan#3874 · Opened Jul 6, 2026
[RL/DeepEP] Generator OOMs in vLLM cudagraph capture on memory-tight MoE (expandable_segments incompatible with DeepEP; gpu_memory_utilization is a no-op) Open

pytorch/torchtitan#3865 · 1 comment · 1 assignee · Opened Jul 6, 2026
DeepEP v2 doesn't work with per-layer compile Open

pytorch/torchtitan#3830 · 2 comments · 1 assignee · Opened Jun 30, 2026
RFC CUDAGraphable GraphPP Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

pytorch/torchtitan#3820 · 2 comments · Opened Jun 29, 2026
TP + MXFP8 crashing with "mat2 must be col_major, got stride (4096, 1)" error Open

module: float8

Difficulty 4/5 3-5 days Newbie friendliness 38/100

pytorch/torchtitan#3817 · 1 comment · Opened Jun 29, 2026
GraphTrainer DSv3-16B: healthy single-node, MI355 FSDP EP=2 parity gap Open

module: torch.compile

Difficulty 5/5 Over a week Newbie friendliness 38/100

pytorch/torchtitan#3816 · 1 comment · Opened Jun 29, 2026
8 GPU Model Tests red on main: gpt-oss PP+FSDP+EP+SACOP MoE _grouped_mm index OOB (CUDA scatter assert == ROCm HSA 0x1016) Open

module: rocm

Difficulty 4/5 3-5 days Newbie friendliness 48/100

pytorch/torchtitan#3761 · 2 comments · Opened Jun 24, 2026
[rl] StickySessionRoutingStrategy can overload a generator for long multi-turn rollouts (no KV-budget awareness / no rebalancing) Open

pytorch/torchtitan#3755 · 2 comments · 2 reactions · 1 assignee · Opened Jun 24, 2026
vllm v2 model runner does not work with VLLMGenerator DP routing Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

pytorch/torchtitan#3752 · Opened Jun 23, 2026
[GraphTrainer] [AutoParallel] Tracker for outstanding issues Open

pytorch/torchtitan#3699 · 1 comment · 1 assignee · Opened Jun 17, 2026
Operator torchtitan.deterministic_topk.default does not have a sharding strategy registered. Open

pytorch/torchtitan#3645 · 1 comment · 1 assignee · Opened Jun 12, 2026
Move pretraining batch shape config out of Trainer.TrainingConfig, into DataLoader.BatchConfig Open

pytorch/torchtitan#3601 · 1 reaction · 1 assignee · Opened Jun 10, 2026
FlexAttention `BlockMask` breaks Pipeline Parallelism with split-backward (zero-bubble / multi) schedules Open

high priority module: flex attention module: pipelining triage review

pytorch/torchtitan#3584 · 7 comments · 1 assignee · Opened Jun 9, 2026
[RL] vLLM attention masks life cycle Open

module: flex attention

pytorch/torchtitan#3572 · 4 comments · 2 assignees · Opened Jun 8, 2026

pytorch/torchtitan

Get new issues in your inbox