Pretraining (with CPUs)

I'm new to deep learning but have some experience with training boosted-decision-trees.

Is this just for fine-tuning or pretraining as well? When I look inside train_gpt2.c I see the first thing it does is it loads weights from a bin file (gpt2_124M.bin). Where did this bin file come from? Is this an official file released by OpenAI? I would like to be able to start from scratch.

I would like to first see how pretraining works, even if it's just a small dataset, and it doesn't need to be GPUs. I would like to start with CPUs first, and maybe add CPU-only nodes that can work on 'parts' of the training.

Start by reading train_gpt2.c, especially where it loads gpt2_124M.bin, and determine how that file is produced. The issue does not identify tests or a specific implementation path; completion would require defining and validating CPU-only pretraining from scratch on a small dataset.

Written by the indexing model from the issue text.