bigscience-workshop / bigscience-workshop/bigscience
How to get train-splits.txt and valid-splits.txt before training tr11-176B-ml
Open
- Dominant language
- Shell
- Stars
- 1k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
* Big Science version: latest
* Python version: 3.8.8
* Operating System: Ubuntu 20.04.5 LTS
### Description
How to get train-splits.txt and valid-splits.txt at Line39 in [train/tr11-176B-ml/tr11-176B-ml.slurm](https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/tr11-176B-ml.slurm#L39
). Thx.
`TRAIN_DATA_PATH=$MEGATRON_DEEPSPEED_REPO/data/train-splits.txt`
`VALID_DATA_PATH=$MEGATRON_DEEPSPEED_REPO/data/valid-splits.txt`
Contributor guide
Assessment
This issue has not been assessed yet.