bigscience-workshop/Megatron-DeepSpeed

View on GitHub

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Stars: 1.4k
Forks: 226
Open beginner issues: 0
Indexed issues: 76

Dominant language: Python
License: No license data
Last GitHub push: Dec 8, 2023
Latest indexed: Sep 14, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: good first issue
PR merge metrics: No merged PRs in 30d

Open All

76 open issues indexed

Issue to gather Fixes + New features to send upstream Open

bigscience-workshop/Megatron-DeepSpeed#10 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Implement Gradient Noise Scale monitoring Open

arch&scale

bigscience-workshop/Megatron-DeepSpeed#33 · 9 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
DeBERTa-like attention mechanism Open

arch&scale enhancement

bigscience-workshop/Megatron-DeepSpeed#49 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Implement the ML Flow experiment tracker Open

🌍 Carbon

bigscience-workshop/Megatron-DeepSpeed#54 · 9 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
recovering from loss spikes strategies Open

bigscience-workshop/Megatron-DeepSpeed#57 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Calling IndexedDatasetBuilder directly with a best_fit datatype fails Open

bigscience-workshop/Megatron-DeepSpeed#67 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[testing] data size / dynamic downloads - test speed and repo bloat Open

bigscience-workshop/Megatron-DeepSpeed#70 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Set up a basic MLflow setup Open

MLFlow

bigscience-workshop/Megatron-DeepSpeed#87 · 1 comment · 0 reactions · 3 assignees · Indexed Sep 14, 2026
How to load a tensor+pipeline parallel checkpoint for inference tasks? Open

bigscience-workshop/Megatron-DeepSpeed#95 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Double counts in parameter count Open

bigscience-workshop/Megatron-DeepSpeed#99 · 2 comments · 0 reactions · 2 assignees · Indexed Sep 14, 2026
Create AWS image with our tools and codebase Open

enhancement

bigscience-workshop/Megatron-DeepSpeed#119 · 9 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Need model size dumped at init Open

Good First Issue

bigscience-workshop/Megatron-DeepSpeed#123 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
adding consistency calculations/checks at init time Open

Good First Issue Good Second Issue

bigscience-workshop/Megatron-DeepSpeed#124 · 11 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Plot scaling laws of our baseline models Open

arch&scale Good First Issue

bigscience-workshop/Megatron-DeepSpeed#125 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
update converted models to include tokenizer files Open

bigscience-workshop/Megatron-DeepSpeed#127 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
is fused layernorm really better? Open

bigscience-workshop/Megatron-DeepSpeed#136 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
clone HF's `GPT2` to create `GPTMeg` with a few tiny changes. Open

Good First Issue Good Second Issue

bigscience-workshop/Megatron-DeepSpeed#138 · 6 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
extract and log grad norm for individual layers Open

Good First Issue

bigscience-workshop/Megatron-DeepSpeed#149 · 10 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[prefixLM] Investigate cuda kernels Open

bigscience-workshop/Megatron-DeepSpeed#153 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[Tensorboard] Log text prediction in evaluation Open

Good First Issue

bigscience-workshop/Megatron-DeepSpeed#163 · 14 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Error in Installation due to circular import Open

bigscience-workshop/Megatron-DeepSpeed#173 · 4 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
Parallelize Meg CUDA Kernel build system Open

Good Difficult Issue Good First Issue

bigscience-workshop/Megatron-DeepSpeed#174 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[deepspeed pipe] expand the partitioning method to support weights Open

Good First Issue

bigscience-workshop/Megatron-DeepSpeed#186 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Fix `sample_idxs_to_text.py` to account for skips Open

bigscience-workshop/Megatron-DeepSpeed#189 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
Avoid re-computing model parameter count every iteration Open

bigscience-workshop/Megatron-DeepSpeed#203 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
Make sure deepspeed powered models are equivalent with their non deepspeed version Open

Good First Issue

bigscience-workshop/Megatron-DeepSpeed#226 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
possibly --skip-train-iteration-range with multiple entries has a bug Open

bigscience-workshop/Megatron-DeepSpeed#230 · 13 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
[new feature] train finishing ETA! Open

Good First Issue

bigscience-workshop/Megatron-DeepSpeed#242 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[new feature] add a `tag` to the arguments to load the checkpoint from a specific step (not necessarly the latest) Open

bigscience-workshop/Megatron-DeepSpeed#279 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
DeepSpeedCheckpoint needs to support bf16 optimizer states. Open

bigscience-workshop/Megatron-DeepSpeed#280 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
DeepSpeed checkpoint performance improvements Open

bigscience-workshop/Megatron-DeepSpeed#312 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Multi-node inference with Bloom: Unhandled CUDA error in ProcessGroupNCCL.cpp (called from all_reduce in torch) Open

bigscience-workshop/Megatron-DeepSpeed#318 · 31 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
where can I download the 176B checkpoint in deepspeed format? Open

bigscience-workshop/Megatron-DeepSpeed#319 · 17 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
"Mask is silently ignored due to the use of a custom kernel" with pretrain_gpt_single_node.sh Open

bigscience-workshop/Megatron-DeepSpeed#320 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Question about downloading checkpoints of 6.3B，2.5B，1.3B Open

bigscience-workshop/Megatron-DeepSpeed#323 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Errors in generation (Bloom) when changing options sampling/use_cache Open

bigscience-workshop/Megatron-DeepSpeed#324 · 29 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Can we also train BLOOM model using tensor using tensor-Parallelism and efficient fused CUDA kernels Open

bigscience-workshop/Megatron-DeepSpeed#334 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Finetuning BLOOM Open

bigscience-workshop/Megatron-DeepSpeed#337 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
About convert deepspeed to deepspeed checkpoint Open

bigscience-workshop/Megatron-DeepSpeed#338 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Installing Apex on Windows Open

bigscience-workshop/Megatron-DeepSpeed#342 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
About reshape deepspeed checkpoint Open

bigscience-workshop/Megatron-DeepSpeed#343 · 20 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
grad norm increase strangely Open

bigscience-workshop/Megatron-DeepSpeed#347 · 12 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Slower inference results for BLOOM fp16 on identical hardware Open

bigscience-workshop/Megatron-DeepSpeed#348 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Load Bloom Optimizer State (i.e. Bloom 1B1) Open

bigscience-workshop/Megatron-DeepSpeed#350 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
deepspeed_to_megatron several issues Open

bigscience-workshop/Megatron-DeepSpeed#355 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
User Warnings for accessing grad attribute of non-leaf Tensors thrown with TP=1 and PP>1 Open

bigscience-workshop/Megatron-DeepSpeed#356 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Fatal error: cuda_fp16.h: No such file or directory on ROCm Open

bigscience-workshop/Megatron-DeepSpeed#360 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Is there any script for pretraining/funting Bloom? Open

bigscience-workshop/Megatron-DeepSpeed#363 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Are there any other layer norm functions, such as RMSNorm or DeepNorm Open

bigscience-workshop/Megatron-DeepSpeed#364 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How to continue pre-training Bloom? Open

bigscience-workshop/Megatron-DeepSpeed#366 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026

bigscience-workshop/Megatron-DeepSpeed

Get new issues in your inbox