bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed

Fix `sample_idxs_to_text.py` to account for skips

Open
#189 0 comments 0 reactions 1 assignee Claimed by @jaketae View on GitHub
Dominant language
Python
Stars
1.4k
Forks
226
PR merge metrics
No merged PRs in 30d

Description

## Motivation
#177 allows for train iterations to be skipped. This functionality is achieved by using a separate internal counter to keep track of skipped iterations instead of tinkering with Meg-DS's internal counters such as `args.iteration`. This introduces a mismatch between the text retrieved from `sample_idxs_to_text.py` and the actual data that was used for training at a given iteration.

## Example
Let's say a user wants to see a dump of data from iterations 10 to 12. If iterations 2 to 4 were skipped in training, then the actual index that we want to retrieve is from 13 to 15.

## Spec

- add support for `--skip-train-iteration-range` from https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/177
- it needs to do the same data skipping code as in the above PR

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.