karpathy/llm.c

View on GitHub

LLM training in simple, raw C/CUDA

Stars: 31k
Forks: 3.8k
Open beginner issues: 0
Indexed issues: 92

Dominant language: Cuda
License: MIT
Last GitHub push: May 10, 2025
Latest indexed: Sep 18, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: good first issue
PR merge metrics: No merged PRs in 30d

Open All

92 open issues indexed

examples for popular models Open

karpathy/llm.c#1 · 4 comments · 4 reactions · 0 assignees · Indexed Sep 18, 2026
Calloc NULL not handled Open

karpathy/llm.c#10 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Alternative tests / consider modularized approach Open

karpathy/llm.c#11 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
write LLVM optimization passes for train_gpt2 Open

karpathy/llm.c#18 · 2 comments · 3 reactions · 0 assignees · Indexed Sep 18, 2026
[Proposal] Implement GaLore trainer Open

karpathy/llm.c#26 · 0 comments · 4 reactions · 0 assignees · Indexed Sep 18, 2026
Why not Mojo? Open

karpathy/llm.c#28 · 13 comments · 4 reactions · 0 assignees · Indexed Sep 18, 2026
Why CUDA when we can SYCL Open

karpathy/llm.c#31 · 3 comments · 32 reactions · 0 assignees · Indexed Sep 18, 2026
Bus ERROR while running `train_gpt2.py` Open

karpathy/llm.c#35 · 14 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Generation error on MPS (Torch >= 2.2.0, MacOS 14.4) Open

karpathy/llm.c#36 · 8 comments · 1 reaction · 0 assignees · Indexed Sep 18, 2026
Suboptimal warp reductions Open

karpathy/llm.c#39 · 0 comments · 5 reactions · 0 assignees · Indexed Sep 18, 2026
Support MPI distributed training Open

karpathy/llm.c#40 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
the provided PTX was compiled with an unsupported toolchain. Open

karpathy/llm.c#63 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
looking forward supporting winx86-msvc Open

karpathy/llm.c#65 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
[build failed]Compiler encountered an internal error Open

karpathy/llm.c#69 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
A file not found error was encountered while compiling Open

karpathy/llm.c#70 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
LOSS MISMATCH AT STEP 0: 2.864161 5.270007 Open

karpathy/llm.c#77 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
AssertionError: Torch not compiled with CUDA enabled Open

karpathy/llm.c#88 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Support older CUDA GPU hardware by default Open

karpathy/llm.c#92 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 18, 2026
output is not consistent when I load the gpt2_124M.bin Open

karpathy/llm.c#93 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 18, 2026
some Rust error Open

karpathy/llm.c#102 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
[CUDA ERROR] at file train_gpt2.cu:693: out of memory Open

karpathy/llm.c#104 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
[CUDA ERROR] at file \llm.c\train_gpt2.cu:405: too many resources requested for launch (old version does not have this issue - fyi) Open

karpathy/llm.c#106 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Little speed up by simple modification is possible Open

karpathy/llm.c#125 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 18, 2026
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. Open

karpathy/llm.c#136 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
The loss doesn't seem to converge after 1000 iterations Open

karpathy/llm.c#138 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
bt-invariant inference Open

feature-request

karpathy/llm.c#146 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Should we use 'T' instead of 'BT' here? I'm not sure either Open

karpathy/llm.c#171 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 18, 2026
CPU: atomicAdd(float*) Open

karpathy/llm.c#176 · 12 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
layernorm_backward.cu: atomicAdd Open

karpathy/llm.c#190 · 0 comments · 2 reactions · 0 assignees · Indexed Sep 18, 2026
Is there a final bin file that can input some question and output some answer? Open

karpathy/llm.c#203 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Input token length question Open

karpathy/llm.c#205 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
test_gpt2.cu correctness bounds tune per-parameter Open

feature-request good first issue

karpathy/llm.c#223 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Error while running "make train_gpt2fp32cu" on Ubuntu Open

karpathy/llm.c#237 · 15 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
init from scratch Open

good first issue

karpathy/llm.c#243 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Splitting cuda dev files to use smaller sizes for cpu validation compared to profiling Open

karpathy/llm.c#244 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
cuda code that approaches cublas performance Open

karpathy/llm.c#255 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 18, 2026
Hardcoded block_size in kernels Open

karpathy/llm.c#261 · 14 comments · 2 reactions · 0 assignees · Indexed Sep 18, 2026
inf loss at big batch Open

bug

karpathy/llm.c#263 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 18, 2026
delete use of cooperative groups in kernels Open

feature-request

karpathy/llm.c#292 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Possible bugs in the data loading functions Open

karpathy/llm.c#321 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 18, 2026
What would be the main design trade-offs when re-implementing in clean modern C++? Open

karpathy/llm.c#354 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
When will llama and other frameworks be supported? Open

karpathy/llm.c#362 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 18, 2026
Assertion `graph->check_support(cudnn_handle).is_good()' failed Open

karpathy/llm.c#366 · 21 comments · 1 reaction · 0 assignees · Indexed Sep 18, 2026
How to do Inference on the trained weight of GPT 2 model after finishing the training on CPU using train_gpt2.py and train_gpt2 ? Open

karpathy/llm.c#372 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 18, 2026
more detailed explanation of Multi GPU Open

karpathy/llm.c#373 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
`make` fails to autodetect GPU compute capability Open

karpathy/llm.c#387 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 18, 2026
Is there a plan to support 8bits (FP8 or INT8)? Open

karpathy/llm.c#391 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026
compute sanitizers Open

karpathy/llm.c#393 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 18, 2026
Broader vendor support for hardware acceleration Open

karpathy/llm.c#400 · 5 comments · 1 reaction · 0 assignees · Indexed Sep 18, 2026
2D and 3D tile divisions so that permutation coordinates can be read from threadIdx and blockIdx Open

karpathy/llm.c#406 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 18, 2026

karpathy/llm.c

Get new issues in your inbox