EleutherAI / EleutherAI/pythia

Provide the shuffled index_mapping npy files for ease of reproducing training data

Open
#153 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.9k
Forks
224
PR merge metrics
No merged PRs in 30d

Description

Hi,

I was wondering can you provide the index_mapping files that is generated by the [GPT2Dataset](https://github.com/EleutherAI/gpt-neox/blob/03186decef022dc35e6adee1a66619968812e0a9/megatron/data/gpt2_dataset.py#L29)? From the construction of gpt2dataset at [here](https://github.com/EleutherAI/gpt-neox/blob/03186decef022dc35e6adee1a66619968812e0a9/megatron/data/gpt2_dataset.py#L158), I can see there are three `npy` index files
```
doc_idx_filename = _filename + "_doc_idx.npy"
sample_idx_filename = _filename + "_sample_idx.npy"
shuffle_idx_filename = _filename + "_shuffle_idx.npy"
```
I was wondering can you provide a copy of these files so that I don't need to regenerate them?

I ask this request because I want to study the influence of the original training data by chunk. I have prepared the pythia-dedup dataset, but I failed to build the environments. After reading the code of [GPT2Dataset](https://github.com/EleutherAI/gpt-neox/blob/03186decef022dc35e6adee1a66619968812e0a9/megatron/data/gpt2_dataset.py#L29), I found that with these index files, I can reproduce the original training data of pythia.

I noticed that you provide the `batch_viewer.py` to check the unshuffled data, but it seems that these data is still different from the actually training data that is fed into the model during the training process.

Thanks

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.