huggingface / huggingface/dataset-viewer

Validate source Parquet files before linking in refs/convert/parquet

Open
#1,504 5 comments 0 reactions 0 assignees View on GitHub
bug P2
Dominant language
Python
Stars
899
Forks
130
Avg merge
9h 26m
Merged PRs (30d)
14

Description

While trying to read a parquet file from dataset revision refs/convert/parquet (generated by datasets-server) with duckdb, it throws the following error:

```
D select * from 'https://huggingface.co/datasets/Pavithra/sampled-code-parrot-train-100k/resolve/refs%2Fconvert%2Fparquet/Pavithra--sampled-code-parrot-train-100k/parquet-train.parquet' limit 10;
Error: Invalid Input Error: No magic bytes found at end of file 'https://huggingface.co/datasets/Pavithra/sampled-code-parrot-train-100k/resolve/refs%2Fconvert%2Fparquet/Pavithra--sampled-code-parrot-train-100k/parquet-train.parquet'

```
Parquet file:
https://huggingface.co/datasets/Pavithra/sampled-code-parrot-train-100k/resolve/refs%2Fconvert%2Fparquet/Pavithra--sampled-code-parrot-train-100k/parquet-train.parquet

Not sure if this is an error in the parquet generation logic. Should we validate non-corrupted files are being pushed to the dataset repo?
I was able to reproduce the same (corrupted error) with polars:
```
Traceback (most recent call last):
File "", line 1, in
File "/home/andrea/.pyenv/versions/3.9.15/lib/python3.9/site-packages/polars/io/parquet/functions.py", line 123, in read_parquet
return pl.DataFrame._read_parquet(
File "/home/andrea/.pyenv/versions/3.9.15/lib/python3.9/site-packages/polars/dataframe/frame.py", line 865, in _read_parquet
self._df = PyDataFrame.read_parquet(
exceptions.ArrowErrorException: ExternalFormat("File out of specification: The file must end with PAR1")

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure against the dataset file at refs/convert/parquet with DuckDB or Polars, confirming the missing PAR1 footer. Trace the parquet generation and linking flow for this revision. Done means corrupted source files are detected before they are linked or pushed, with a clear failure or prevention path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.