EleutherAI / EleutherAI/pythia

seed2 shuffle_idx.npy corrupted

Open
#190 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.9k
Forks
224
PR merge metrics
No merged PRs in 30d

Description

Hi, I think the shuffle_idx.npy for seed2 is corrupted.

Can someone else reproduce these results?
(a copy of the issue exists on huggingface: https://huggingface.co/datasets/EleutherAI/pile-preshuffled-seeds/discussions/1)

Expected:
The shuffle_idx file is a permutation, so every index should appear once.

Observed:
All entries between index 12354528 and index 162164735 are zero (about 92%).

```python
import numpy as np

path = "seed2/pile_20B_tokenizer_text_document_train_0_indexmap_147164160ns_2048sl_2s_shuffle_idx.npy"

shuffle = np.load(path, allow_pickle=True)
u, c = np.unique(shuffle, return_counts=True)
print(u[:10])
# out: [ 0 12 21 25 38 44 68 71 82 90]
print(c[:10])
# out: [149810208 1 1 1 ...

first, last = np.where(shuffle == 0)[0][[0, -1]]

np.unique(shuffle[first: last+1], return_counts=True)
# out: (array([0], dtype=uint32), array([149810207]))
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.