bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
Fix `sample_idxs_to_text.py` to account for skips
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
## Motivation
#177 allows for train iterations to be skipped. This functionality is achieved by using a separate internal counter to keep track of skipped iterations instead of tinkering with Meg-DS's internal counters such as `args.iteration`. This introduces a mismatch between the text retrieved from `sample_idxs_to_text.py` and the actual data that was used for training at a given iteration.
## Example
Let's say a user wants to see a dump of data from iterations 10 to 12. If iterations 2 to 4 were skipped in training, then the actual index that we want to retrieve is from 13 to 15.
## Spec
- add support for `--skip-train-iteration-range` from https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/177
- it needs to do the same data skipping code as in the above PR
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.