bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
Plot scaling laws of our baseline models
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
For our three baselines on different datasets (OSCAR, C4, The Pile), we would like to plot scaling laws and retrieve their coefficients. Specifically, we are looking to reproduce Figure 1 of [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361 ).
The TensorBoard data for the baseline runs can be retrieved on the [Big Science space](https://huggingface.co/bigscience) on HuggingFace: it's the `tr3` runs with `tensorboard` in their name. The naming scheme (`tr3b`, `tr3c`, etc.) is explained [here](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr3-1B3-baseline).
For C4, we have a XL, L, and M model (`tr3`, `tr3c`, `tr3c`) with short warm-up. For OSCAR and The Pile, we have an XL, L, M, and S model (`tr3d`, `tr3g`, `tr3h`, `tr3i` and `tr3`, `tr3j`, `tr3k`, `tr3l`). For OSCAR, we can should also add the 13B run to see if the fits hold (that's `tr1-13B`).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.