deepspeedai/DeepSpeed

View on GitHub

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Stars: 43.1k
Forks: 5k
Open beginner issues: 5
Indexed issues: 1.2k
Avg merge: 4d 15h
Merged PRs (30d): 112

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 20, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

1,181 open issues indexed

[REQUEST] NPU: implement graph operations for short kernel sequences (follow-up to #4318) Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 78/100

deepspeedai/DeepSpeed#8604 · Opened Sep 20, 2026
[BUG]_prepare_fp32_grad_for_sub_group clears every parameter's gradient once per sub-group Open

bug training

deepspeedai/DeepSpeed#8586 · 1 assignee · Opened Sep 18, 2026
init_inference rejects dtype=torch.int8 on every accelerator Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

deepspeedai/DeepSpeed#8577 · Opened Sep 18, 2026
[BUG] Runtime configuration validation is bypassed under python -O Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

deepspeedai/DeepSpeed#8572 · 1 comment · Opened Sep 17, 2026
2 tests fail: assert isinstance(model.llm[0], DistributedAttention) Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

deepspeedai/DeepSpeed#8554 · Opened Sep 16, 2026
[BUG] AutoTP checkpoint load raises AttributeError on a model containing nn.InstanceNorm1d Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

deepspeedai/DeepSpeed#8514 · Opened Sep 14, 2026
[BUG] Python checkpoint writers fail to reload full ZeRO-3 state with default torch.load Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

deepspeedai/DeepSpeed#8500 · Opened Sep 13, 2026
[BUG] ZeRO-3 ignores checkpoint tag when load_optimizer_states=False Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

deepspeedai/DeepSpeed#8499 · Opened Sep 13, 2026
Deprecate unused DeepSpeed features Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

deepspeedai/DeepSpeed#8489 · 6 comments · Opened Sep 11, 2026
[BUG] Memory leak in loss backward when enabled torch activation checkpoint & zero3 & single-model-multi-branch network Open

bug training

Difficulty 4/5 3-5 days Newbie friendliness 48/100

deepspeedai/DeepSpeed#8481 · Opened Sep 11, 2026
Can't install in Python 3.14.6 with GCC Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

deepspeedai/DeepSpeed#8480 · 1 comment · Opened Sep 11, 2026
[Bug] AutoEP FP32 gradient clipping averages norms across ranks with different expert ownership Open

bug training

Difficulty 4/5 3-5 days Newbie friendliness 32/100

deepspeedai/DeepSpeed#8475 · 1 comment · Opened Sep 10, 2026
[RFC] Enable Correct and High-Performance Muon Optimizer under ZeRO CPU Offload (Stage 1/2/3) Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

deepspeedai/DeepSpeed#8463 · 1 reaction · Opened Sep 8, 2026
[FEATURE] Align decode fronts and trim dead zones in continuous batching Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

deepspeedai/DeepSpeed#8462 · Opened Sep 8, 2026
[BUG] Do not silently ignore use_shared_prefill with continuous batching Open

bug training

deepspeedai/DeepSpeed#8458 · 2 comments · 1 assignee · Opened Sep 8, 2026
AutoTP loads module weights by hardcoded class name, so 219 of 233 transformers norm classes are silently left on meta Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

deepspeedai/DeepSpeed#8447 · Opened Sep 6, 2026
ZeRO-3 applies Muon's Newton-Schulz once per micro-batch, so gradient accumulation changes the optimizer Open

Difficulty 4/5 3-5 days Newbie friendliness 50/100

deepspeedai/DeepSpeed#8443 · 4 comments · Opened Sep 6, 2026
Default gradient_clipping divides every Muon update by its own norm, shrinking the step by a model-sized factor Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

deepspeedai/DeepSpeed#8439 · 1 comment · Opened Sep 6, 2026
Muon under AutoTP orthogonalizes each shard, so the update depends on the TP degree Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

deepspeedai/DeepSpeed#8437 · 1 comment · Opened Sep 6, 2026
[Per-Head Muon] Head geometry that lives outside num_attention_heads/head_dim: Kimi-K3 KDA layers and GLM-5.2's DSA indexer Open

deepspeedai/DeepSpeed#8420 · 3 comments · 1 assignee · Opened Sep 5, 2026
ZeRO skips the optimizer step on gradient overflow without logging anything Open

deepspeedai/DeepSpeed#8415 · 1 reaction · 1 assignee · Opened Sep 4, 2026
`_cast_module_mixed_precision` casts every dtype torch calls floating point, corrupting FP8/MX/NVFP4 parameters Open

deepspeedai/DeepSpeed#8414 · 1 reaction · 1 assignee · Opened Sep 4, 2026
[BUG] Curriculum data sampler checkpoints the global numpy RNG, not its own, so resume replays the sampling stream Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

deepspeedai/DeepSpeed#8405 · Opened Sep 3, 2026
ZeRO-3 HybridMegatron QKV layout mutations are not persisted Open

Difficulty 3/5 1-2 days Newbie friendliness 74/100

deepspeedai/DeepSpeed#8391 · Opened Sep 2, 2026
[BUG] set_norm_for_param_grad_in_gpu's grad_accum fallback is dead in both accumulation modes Open

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

deepspeedai/DeepSpeed#8371 · Opened Aug 31, 2026
[Feature] Per-Head Muon: per-head Newton-Schulz orthogonalization for attention projections Open

enhancement training

Difficulty 5/5 Over a week Newbie friendliness 45/100

deepspeedai/DeepSpeed#8367 · 5 comments · 4 reactions · Opened Aug 30, 2026
ZeRO-1/2 average_tensor serializes backward compute behind the preceding reduction Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

deepspeedai/DeepSpeed#8364 · 4 comments · Opened Aug 30, 2026
[BUG]Unfreed parameters and unnecessary peak memory in autoEP Open

bug training

deepspeedai/DeepSpeed#8353 · 1 comment · 1 assignee · Opened Aug 29, 2026
[BUG] DeepSpeed generation fails with a CPU/CUDA device mismatch when keep-module-on-host is enabled Open

bug inference

Difficulty 4/5 3-5 days Newbie friendliness 52/100

deepspeedai/DeepSpeed#8338 · 1 comment · Opened Aug 28, 2026
HybridEngineRollout: synchronized early stopping across ZeRO-3 data-parallel ranks Open

deepspeedai/DeepSpeed#8321 · 5 comments · 1 assignee · Opened Aug 26, 2026
[REQUEST] XPU: register native pinned host memory with the device runtime (follow-up to #8283) Open

deepspeedai/DeepSpeed#8315 · 1 comment · 2 reactions · 1 assignee · Opened Aug 25, 2026
HF transformers main injects 'embedding_rowwise' into tp_plan for tied-embedding models; AutoTP rejects the whole plan Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

deepspeedai/DeepSpeed#8290 · 3 comments · 1 reaction · Opened Aug 22, 2026
Define affine IR for checkpointing Open

enhancement

deepspeedai/DeepSpeed#8252 · 15 comments · 1 reaction · 1 assignee · Opened Aug 14, 2026
[RFC] Universal checkpoint: non-semantic per-parameter geometric shard map Open

deepspeedai/DeepSpeed#8230 · 10 comments · 1 reaction · 1 assignee · Opened Aug 7, 2026
[REQUEST] OPSD Profile and improve HybridEngine rollout performance Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 45/100

deepspeedai/DeepSpeed#8197 · 17 comments · 1 reaction · Opened Jul 31, 2026
[REQUEST] Dynamic/Adaptive Prefetching Window for ZeRO-3 Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

deepspeedai/DeepSpeed#8193 · 3 comments · 1 reaction · Opened Jul 29, 2026
[RFC] Support sharding LM heads and adopting Online Softmax Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

deepspeedai/DeepSpeed#8173 · 5 comments · 2 reactions · Opened Jul 24, 2026
Add logging support for better observability Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

deepspeedai/DeepSpeed#8140 · 1 comment · Opened Jul 14, 2026
DeepSpeed - Q3 Roadmap Open

deepspeedai/DeepSpeed#8104 · 6 comments · 9 reactions · 5 assignees · Opened Jun 30, 2026
Consider adding a changelog to track version history Open

Difficulty 1/5 1-3 hours Newbie friendliness 68/100

deepspeedai/DeepSpeed#8101 · 1 comment · Opened Jun 30, 2026
Exception type too broad in error handling Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

deepspeedai/DeepSpeed#8093 · 1 comment · Opened Jun 26, 2026
zero.Init partitioning of large fused MoE-expert tensors spikes a single GPU (transient full materialization) -> OOM during load even when the sharded model fits Open

Difficulty 5/5 Over a week Newbie friendliness 45/100

deepspeedai/DeepSpeed#8085 · 4 comments · Opened Jun 22, 2026
Add type hints to public API functions Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

deepspeedai/DeepSpeed#8074 · 1 comment · Opened Jun 17, 2026
EvoformerAttention should auto-detect CUTLASS instead of requiring CUTLASS_PATH Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

deepspeedai/DeepSpeed#8001 · 1 comment · Opened May 8, 2026
[BUG]Loss connection to ranks Open

bug training

Difficulty 4/5 3-5 days Newbie friendliness 45/100

deepspeedai/DeepSpeed#7991 · 4 comments · Opened May 4, 2026
[REQUEST] Add Trackio as a New Backend for Experiment Monitoring Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 55/100

deepspeedai/DeepSpeed#7964 · 4 comments · Opened Apr 9, 2026
[BUG] crashed training gpt model using 3d parallelization Open

bug training

Difficulty 4/5 3-5 days Newbie friendliness 35/100

deepspeedai/DeepSpeed#7922 · 2 comments · Opened Mar 25, 2026
[REQUEST] Put NVTX instrumentation in its own domain (and potentially categories) Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

deepspeedai/DeepSpeed#7912 · 3 comments · Opened Mar 18, 2026
Trouble running tests: Failed to find precompiled modules Open

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

deepspeedai/DeepSpeed#7909 · 1 comment · 1 reaction · Opened Mar 17, 2026
Deepspeed zero 3 seems to save empty adapter_model.safetensors Open

Difficulty 4/5 3-5 days Newbie friendliness 28/100

deepspeedai/DeepSpeed#7900 · 1 comment · Opened Mar 13, 2026

deepspeedai/DeepSpeed

Get new issues in your inbox