bigscience-workshop / bigscience-workshop/data-preparation

Mismatch of the Available Data Quantity on Huggingface

Open
#37 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
318
Forks
42
PR merge metrics
No merged PRs in 30d

Description

I tried to download English part of Roots these days.
According to the paper, there are 484,953,009,124 bytes of English data.
However, after downloading all roots-related datasets on [huggingface](https://huggingface.co/datasets?language=language:en&sort=downloads&search=bigscience-data%2F) by filtering, I found there is only about 43.8 GB data.
I wonder how to explain the difference?
Are those huggingface datasets only a subset of Roots?
Are those huggingface datasets processed Roots so that the quantity shrinks from 480 GB to 43.8 GB?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the paper's reported 484,953,009,124 English-data bytes with the approximately 43.8 GB available from the ROOTS-related Hugging Face datasets linked in the issue. Check whether those datasets are a subset or processed version of ROOTS; done means documenting the explanation and the scope of the published datasets.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface
Domain
data
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.