bigscience-workshop / bigscience-workshop/multilingual-modeling
Adding Language specific validation sets to deepspeed
- Dominant language
- Python
- Stars
- 75
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
The idea of this issue to modify the [megatron-deepspeed](https://github.com/bigscience-workshop/Megatron-DeepSpeed) repository code that we use for training all models. In order to track the progress of validation loss on several validaiton sets separately. This would allow us to track the progress of training independtly on separate languages.
Currently, the validation loss is calculated on a single validation set that includes the same language combination as the training data. ([see here](https://huggingface.co/bigscience/tr1-13B-tensorboard/tensorboard) 13B param model training on tensorboard)

### Useful pointers
- How datasets are loaded in model pre-training [here](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/3c9d748bfb15f1ae9ffb09f7c13fc06ad2a169d4/pretrain_gpt.py#L25)
- Dataset loader for GPT [here](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/63671cb474fd08e0e339836abdf0637f93b535fd/megatron/data/gpt_dataset.py)
- Validation step execution [here](https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/3c9d748bfb15f1ae9ffb09f7c13fc06ad2a169d4/megatron/training.py#L712)
## Progress
- Forked deepspeed where all development happens (ask @hadyelsahar for invitation) [here](https://github.com/hadyelsahar/Megatron-DeepSpeed)
- Pull request: https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/97
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.