huggingface / huggingface/dataset-viewer

Store which splits are partial and which are complete

Open
#2,809 0 comments 0 reactions 0 assignees View on GitHub
improvement / optimization P2
Dominant language
Python
Stars
900
Forks
131
Avg merge
9h 26m
Merged PRs (30d)
14

Description

In each step, we should store if we truncated the data or not.

Currently, `config-parquet-and-info` only stores the fact that some of the splits have been partially converted to parquet, but not the list of them.

We want to have the info for each split.

The same goes when we convert to duckdb, and when we compute the statistics. It should be the case on each truncation, so that we can show trustable information in the viewer.

Related issues and discussions:
- https://github.com/huggingface/dataset-viewer/issues/1532
- https://github.com/huggingface-internal/moon-landing/issues/9429
- https://huggingface.slack.com/archives/C0311GZ7R6K/p1715339968924859 for example:
> There i a new dataset https://huggingface.co/datasets/H-D-T/Buzz with 31M samples but the viewer only shows 3.1M. It looks like it only loads the first json file instead of all batches. That lead me and other wrongly believing its 3.1M instead of 31M (edited)
- https://huggingface.slack.com/archives/C04L6P8KNQ5/p1710947648069719
> On another subject, we noticed that the stats showed for datasets like https://huggingface.co/datasets/bigcode/the-stack-v2 are computed only on the first 5GB but the Viewer doesn't show this info anywhere (could be very discrete like on hover or using a small "(i)" for information)

Contributor guide

Open the contributing guide

Research direction

Start by tracing how `config-parquet-and-info` records partial conversion, then follow the corresponding DuckDB conversion and statistics steps. Identify where each truncation decision is made and how the viewer consumes this information. Done means each split is marked partial or complete for parquet conversion, DuckDB conversion, and statistics so the viewer can show trustworthy coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.