EleutherAI / EleutherAI/pythia
Reading data is slowly！
Open
- Dominant language
- Jupyter Notebook
- Stars
- 2.9k
- Forks
- 224
- PR merge metrics
- No merged PRs in 30d
Description
I followed readme：
```
git lfs clone https://huggingface.co/datasets/EleutherAI/pythia_deduped_pile_idxmaps
python utils/unshard_memmap.py --input_file ./pythia_deduped_pile_idxmaps/pile_0.87_deduped_text_document-00000-of-00082.bin --num_shards 83 --output_dir ./pythia_pile_idxmaps/
```
I got a 600+G file, and then I used gpt-neox's dataloader to read the data, which was very slow. It takes about 6s to read 2048-length pieces of data. May I ask why?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.