bigscience-workshop/Megatron-DeepSpeed

View on GitHub

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Stars: 1.4k
Forks: 226
Open beginner issues: 0
Indexed issues: 76

Dominant language: Python
License: No license data
Last GitHub push: Dec 8, 2023
Latest indexed: Sep 14, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: good first issue
PR merge metrics: No merged PRs in 30d

Open All

76 open issues indexed

Exception: cuda rng state model-parallel-rng is not added Open

bigscience-workshop/Megatron-DeepSpeed#369 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How to preprocess data for t5 model? Open

bigscience-workshop/Megatron-DeepSpeed#371 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
how to pretrain t5-lm adapted? Open

bigscience-workshop/Megatron-DeepSpeed#372 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Can I use python only apex for gpt_pretrain? Open

bigscience-workshop/Megatron-DeepSpeed#373 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
how to frozen some layers of GPT, only fintune last k layers? Open

bigscience-workshop/Megatron-DeepSpeed#375 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
how to do prompt learning with bloom? Open

bigscience-workshop/Megatron-DeepSpeed#376 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How can we access to the gradients while the model is training? Open

bigscience-workshop/Megatron-DeepSpeed#377 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
upgrade megatron-lm Open

bigscience-workshop/Megatron-DeepSpeed#378 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
The given group does not exist pytorch Open

bigscience-workshop/Megatron-DeepSpeed#379 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Megatron-DeepSpeed only applies to specific models? Open

bigscience-workshop/Megatron-DeepSpeed#381 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Help me, I'm dying soon，error: command '/opt/rh/devtoolset-7/root/usr/bin/gcc' failed with exit code 1 error: subprocess-exited-with-error Open

bigscience-workshop/Megatron-DeepSpeed#382 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
pip install -e . failed with ModuleNotFoundError: No module named 'torch' Open

bigscience-workshop/Megatron-DeepSpeed#383 · 2 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
How to properly use Flops Profiler with pipelined parallelism? Open

bigscience-workshop/Megatron-DeepSpeed#385 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
hello， I meet a problem Open

bigscience-workshop/Megatron-DeepSpeed#386 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
RuntimeError: Error building extension 'scaled_upper_triang_masked_softmax_cuda' Open

bigscience-workshop/Megatron-DeepSpeed#387 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Question about ds to universal Open

bigscience-workshop/Megatron-DeepSpeed#388 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
questions about inconsistent evaluation result Open

bigscience-workshop/Megatron-DeepSpeed#392 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Question about the implementation of mpu.cross_entropy when using tensor parallel Open

bigscience-workshop/Megatron-DeepSpeed#394 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
The difference between zero-3 and megatron with zero-2 Open

bigscience-workshop/Megatron-DeepSpeed#395 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
the traing log like this is Normal？ I do not find loss in the logs, and what does the "grad norm: nan" mean? Open

bigscience-workshop/Megatron-DeepSpeed#396 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Hello, can Megatron-DeepSpeed pre-train llama2? Open

bigscience-workshop/Megatron-DeepSpeed#398 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Is this assertion for mask wrong? Open

bigscience-workshop/Megatron-DeepSpeed#400 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Hello, what version of the megatron-lm library is your code modified? Open

bigscience-workshop/Megatron-DeepSpeed#401 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
How can I set recomputation-granularity,like selective or full? Open

bigscience-workshop/Megatron-DeepSpeed#403 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
BUG: training gpt2 with pp=2 error:list index out of range Open

bigscience-workshop/Megatron-DeepSpeed#406 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
can we [add new feature ]support zero=2,3 with --tensor-model-parallel-size 2 --pipeline-model-parallel-size 2 for pretrain-gpt2? Open

bigscience-workshop/Megatron-DeepSpeed#407 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026

bigscience-workshop/Megatron-DeepSpeed

Get new issues in your inbox