bigscience-workshop / bigscience-workshop/data-preparation
Mismatch of the Available Data Quantity on Huggingface
- Dominant language
- Jupyter Notebook
- Stars
- 318
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
I tried to download English part of Roots these days.
According to the paper, there are 484,953,009,124 bytes of English data.
However, after downloading all roots-related datasets on [huggingface](https://huggingface.co/datasets?language=language:en&sort=downloads&search=bigscience-data%2F) by filtering, I found there is only about 43.8 GB data.
I wonder how to explain the difference?
Are those huggingface datasets only a subset of Roots?
Are those huggingface datasets processed Roots so that the quantity shrinks from 480 GB to 43.8 GB?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the paper's reported 484,953,009,124 English-data bytes with the approximately 43.8 GB available from the ROOTS-related Hugging Face datasets linked in the issue. Check whether those datasets are a subset or processed version of ROOTS; done means documenting the explanation and the scope of the published datasets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface
- Domain
- data
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100