bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed

Plot scaling laws of our baseline models

Open
#125 2 comments 0 reactions 1 assignee Claimed by @slippylolo View on GitHub
arch&scale Good First Issue
Dominant language
Python
Stars
1.4k
Forks
226
PR merge metrics
No merged PRs in 30d

Description

For our three baselines on different datasets (OSCAR, C4, The Pile), we would like to plot scaling laws and retrieve their coefficients. Specifically, we are looking to reproduce Figure 1 of [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361 ).

The TensorBoard data for the baseline runs can be retrieved on the [Big Science space](https://huggingface.co/bigscience) on HuggingFace: it's the `tr3` runs with `tensorboard` in their name. The naming scheme (`tr3b`, `tr3c`, etc.) is explained [here](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr3-1B3-baseline).
For C4, we have a XL, L, and M model (`tr3`, `tr3c`, `tr3c`) with short warm-up. For OSCAR and The Pile, we have an XL, L, M, and S model (`tr3d`, `tr3g`, `tr3h`, `tr3i` and `tr3`, `tr3j`, `tr3k`, `tr3l`). For OSCAR, we can should also add the 13B run to see if the fits hold (that's `tr1-13B`).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.