TODO improving nanoGPT training

- [x] Look at mixed precision, check lr decay, check the compile arguments (see #7 )
- [x] **Reaching 3.28:** Making the code more similar to this version of [`modded_nanogpt`](https://github.com/KellerJordan/modded-nanogpt/blob/844e5fdb2334ff83324e6f1f900ce443dd9e1226/train_gpt2.py) (@tomMoral )
- [x] Get some plot with the number of processed tokens as [this one](https://x.com/kellerjordan0/status/1844820919061287009/photo/1) (@tomMoral )
- [x] Take a look at [EPFL-scion](https://github.com/LIONS-EPFL/scion/blob/main/examples/modded-nanogpt/train_gpt_scion.py) code which is easier and reaches 3.33 in 5100 iterations
- [ ] Try to use the ZeroRedundancyOptimizer (@tomMoral )
- [x] **Adding SciOn:** we can look at [this code](https://github.com/LIONS-EPFL/scion/blob/main/examples/modded-nanogpt/train_gpt_scion.py#L469) (@tonysf, #6 )
- [x] **Adding muOn:** we can look at [this code](https://github.com/KellerJordan/modded-nanogpt/blob/master/train_gpt.py#L136) (@tonysf )
- [x] Delete the PR #8 which seems in bad state
- [x] Implement SOAP (@svaiter, #11)
- [ ] Implement Shampoo (@tonysf )

Start by reading the linked modded_nanogpt train_gpt2.py and SciOn train_gpt_scion.py examples, then inspect this repository's training entry point and optimizer setup. The remaining work is to evaluate or implement ZeroRedundancyOptimizer and Shampoo; completion should be demonstrated by both unchecked items being resolved with corresponding training results.

Written by the indexing model from the issue text.