EleutherAI/gpt-neox

View on GitHub

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

Stars: 7.5k
Forks: 1.1k
Open beginner issues: 0
Indexed issues: 69
Avg merge: 1m
Merged PRs (30d): 1

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 4, 2026
Latest indexed: Sep 15, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: good first issue help wanted

Open All

69 open issues indexed

loss stuck in overflow for RPE position embedding together with sparse attention Open

bug

EleutherAI/gpt-neox#292 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Add shampoo optimizer Open

feature request

EleutherAI/gpt-neox#314 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Increase Documentation Coverage Open

feature request

EleutherAI/gpt-neox#458 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Hosted Github Runners for CI Open

feature request

EleutherAI/gpt-neox#531 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
ONNX Export / Inference Engine Open

feature request

EleutherAI/gpt-neox#537 · 0 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
Fine-tuning GPT-NeoX doesn't work (for many scenarios) with the 16-bit stage-0 optimizer Open

bug

EleutherAI/gpt-neox#568 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Introduce improvements from OSLO Open

feature request

EleutherAI/gpt-neox#571 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Migrate tensor parallelism code to use OSLO Open

feature request oslo

EleutherAI/gpt-neox#578 · 7 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
CUDA Out of Memory for 20B Model on 2 A100 40GB GPUs Open

EleutherAI/gpt-neox#606 · 7 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
OOM issues running inference with large contexts on 2x3090 system Open

bug

EleutherAI/gpt-neox#631 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
getting error saying __init__ () got an unexpected keyword argument 'checkpointable_layers' when i started training Open

bug

EleutherAI/gpt-neox#632 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Text generation yields different outputs despite temperature = 0.0 Open

bug good first issue

EleutherAI/gpt-neox#643 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
RuntimeError: Error(s) in loading state_dict for EmbeddingPipe: size mismatch for word_embeddings.weight Open

bug good first issue help wanted

EleutherAI/gpt-neox#645 · 9 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Training speed in bf16 mode is slow. Open

bug

EleutherAI/gpt-neox#660 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Interactive inference fails Open

bug

EleutherAI/gpt-neox#699 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Package Megatron Open

feature request

EleutherAI/gpt-neox#708 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
In interactive mode prompt length more than one word causes to crash Open

bug deprioritized

EleutherAI/gpt-neox#758 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Multi-node training without shared memory Open

deprioritized feature request

EleutherAI/gpt-neox#765 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Fine-tuning 20B model doesn't seem to work Open

bug deprioritized

EleutherAI/gpt-neox#767 · 17 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Unable to load model checkpoint with model parallelism Open

feature request

EleutherAI/gpt-neox#773 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Cannot load the checkpoint Open

bug

EleutherAI/gpt-neox#782 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Implement Prefix-LM attention masking Open

feature request

EleutherAI/gpt-neox#805 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Finetuning loss explode when not loading deepspeed zero optimal states Open

bug

EleutherAI/gpt-neox#843 · 9 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Investigate DeepSpeed Inference Open

feature request good first issue

EleutherAI/gpt-neox#845 · 7 comments · 0 reactions · 2 assignees · Indexed Sep 15, 2026
block-sparse flash attention support Open

feature request good first issue

EleutherAI/gpt-neox#851 · 4 comments · 4 reactions · 1 assignee · Indexed Sep 15, 2026
Add support for pytorch 2.0 ? Open

deprioritized feature request

EleutherAI/gpt-neox#858 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
DeepSpeed Sparse Attention is Broken Open

bug

EleutherAI/gpt-neox#863 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
OOM error when training on a 220G Memory machine with 8 V100. Open

feature request

EleutherAI/gpt-neox#867 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Fine-tuning gpt-neox on 8 A100s Open

feature request

EleutherAI/gpt-neox#892 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Add StableLM as an example to the README Open

documentation

EleutherAI/gpt-neox#896 · 3 comments · 3 reactions · 1 assignee · Indexed Sep 15, 2026
20B pretrained model inference OOM on 8xA100 40GB Open

bug good first issue

EleutherAI/gpt-neox#901 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Problems on generating with llama model Open

EleutherAI/gpt-neox#921 · 21 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Can't finetune 20B model from slim weights with zero optimizer enabled Open

bug

EleutherAI/gpt-neox#926 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
The plot got from muP coord_check seems not horizontal, which may indicates there exits a bug in the muP implementation? Open

bug

EleutherAI/gpt-neox#956 · 11 comments · 1 reaction · 1 assignee · Indexed Sep 15, 2026
Bias weights are multi-added when using `gpt_j_residual` in model-parallel execution Open

bug good first issue

EleutherAI/gpt-neox#962 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Why we need to average LayerNorm values over mp ranks when converting to HFformat checkpoint? Open

EleutherAI/gpt-neox#983 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
How to preserve Pythia's sampling order but for different batch size. Open

bug

EleutherAI/gpt-neox#984 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
AssertionError: zero stage 1 requires an optimizer Open

bug good first issue help wanted

EleutherAI/gpt-neox#987 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Convert HF Llama Checkpoints to Neox Checkpoints Open

feature request

EleutherAI/gpt-neox#994 · 2 comments · 3 reactions · 0 assignees · Indexed Sep 15, 2026
[BUG] Inconsistent loss between `overlap_comm=true` and `overlap_comm=false` Open

bug

EleutherAI/gpt-neox#1004 · 5 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Support for Mosaic Models Open

feature request

EleutherAI/gpt-neox#1057 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Interoperability and GPT-NeoX Open

documentation question

EleutherAI/gpt-neox#1058 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
[BUG?] Higher "gradient_accumulation_steps" still increases memory usage a lot Open

bug

EleutherAI/gpt-neox#1123 · 3 comments · 1 reaction · 1 assignee · Indexed Sep 15, 2026
Tests fail when run with pytest --forked Open

bug

EleutherAI/gpt-neox#1132 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
PyTorch Lightning Fused optimizer step Open

feature request

EleutherAI/gpt-neox#1160 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
My servers used for multi-node training do not have ssh. How can I launch multi-node training using the torchrun command? Open

feature request

EleutherAI/gpt-neox#1203 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
LoRA Support Open

feature request

EleutherAI/gpt-neox#1204 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Cannot convert neox model to HF Open

bug

EleutherAI/gpt-neox#1231 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
batch_input and elapsed time per iteration suddenly slow down during model training Open

bug

EleutherAI/gpt-neox#1248 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Assertion Error when Setting pipe_parallel_size or model_parallel_size in GPT-NeoX Open

bug

EleutherAI/gpt-neox#1251 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026

EleutherAI/gpt-neox

Get new issues in your inbox