bigscience-workshop / bigscience-workshop/multilingual-modeling

Adding Language specific validation sets to deepspeed

Open
#1 4 comments 0 reactions 1 assignee Claimed by @hadyelsahar View on GitHub
Dominant language
Python
Stars
75
Forks
17
PR merge metrics
No merged PRs in 30d

Description

The idea of this issue to modify the [megatron-deepspeed](https://github.com/bigscience-workshop/Megatron-DeepSpeed) repository code that we use for training all models. In order to track the progress of validation loss on several validaiton sets separately. This would allow us to track the progress of training independtly on separate languages.

Currently, the validation loss is calculated on a single validation set that includes the same language combination as the training data. ([see here](https://huggingface.co/bigscience/tr1-13B-tensorboard/tensorboard) 13B param model training on tensorboard)

![image](https://user-images.githubusercontent.com/1453243/132491704-767d76fe-250c-486f-b7e1-91518f7d2d4d.png)

### Useful pointers
- How datasets are loaded in model pre-training [here](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/3c9d748bfb15f1ae9ffb09f7c13fc06ad2a169d4/pretrain_gpt.py#L25)
- Dataset loader for GPT [here](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/63671cb474fd08e0e339836abdf0637f93b535fd/megatron/data/gpt_dataset.py)
- Validation step execution [here](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/3c9d748bfb15f1ae9ffb09f7c13fc06ad2a169d4/megatron/training.py#L712)

## Progress
- Forked deepspeed where all development happens (ask @hadyelsahar for invitation) [here](https://github.com/hadyelsahar/Megatron-DeepSpeed)
- Pull request: https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/97

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.