bigscience-workshop / bigscience-workshop/xmtf
Quesiton about MTFDataset
Open
- Dominant language
- Jupyter Notebook
- Stars
- 534
- Forks
- 43
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/main/megatron/data/mtf_dataset.py#L34
The MTFDataset class take `documents` as arguments, but didn't use it(except in assert statement).
I think `documents` is train/valid/test split index, is it ok to ignore `documents`?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.