huggingface / huggingface/dataset-viewer
Store which splits are partial and which are complete
- Dominant language
- Python
- Stars
- 900
- Forks
- 131
- Avg merge
- 9h 26m
- Merged PRs (30d)
- 14
Description
In each step, we should store if we truncated the data or not.
Currently, `config-parquet-and-info` only stores the fact that some of the splits have been partially converted to parquet, but not the list of them.
We want to have the info for each split.
The same goes when we convert to duckdb, and when we compute the statistics. It should be the case on each truncation, so that we can show trustable information in the viewer.
Related issues and discussions:
- https://github.com/huggingface/dataset-viewer/issues/1532
- https://github.com/huggingface-internal/moon-landing/issues/9429
- https://huggingface.slack.com/archives/C0311GZ7R6K/p1715339968924859 for example:
> There i a new dataset https://huggingface.co/datasets/H-D-T/Buzz with 31M samples but the viewer only shows 3.1M. It looks like it only loads the first json file instead of all batches. That lead me and other wrongly believing its 3.1M instead of 31M (edited)
- https://huggingface.slack.com/archives/C04L6P8KNQ5/p1710947648069719
> On another subject, we noticed that the stats showed for datasets like https://huggingface.co/datasets/bigcode/the-stack-v2 are computed only on the first 5GB but the Viewer doesn't show this info anywhere (could be very discrete like on hover or using a small "(i)" for information)
Contributor guide
Research direction
Start by tracing how `config-parquet-and-info` records partial conversion, then follow the corresponding DuckDB conversion and statistics steps. Identify where each truncation decision is made and how the viewer consumes this information. Done means each split is marked partial or complete for parquet conversion, DuckDB conversion, and statistics so the viewer can show trustworthy coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100