huggingface / huggingface/datasets

load_dataset silently accepts out-of-range ClassLabel indices when the source dtype already matches

Open
#8,596 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

Loading a file whose parsed Arrow dtype already matches a declared `ClassLabel`'s storage type (`int64`) skips the range check entirely. Out-of-range indices load with no error and no warning, and only surface later at `int2str` time or as silently wrong labels during training.

```python
from datasets import Dataset, Features, ClassLabel
import json, tempfile, os

with tempfile.TemporaryDirectory() as d:
p = os.path.join(d, "data.jsonl")
with open(p, "w") as f:
f.write(json.dumps({"label": 5}) + "\n")
f.write(json.dumps({"label": 1}) + "\n")
ds = Dataset.from_json(p, features=Features({"label": ClassLabel(names=["neg", "pos", "oth"])}))
print(ds["label"]) # [5, 1], no error, only 3 classes declared
```

Same result with Parquet:

```python
import pyarrow as pa, pyarrow.parquet as pq
from datasets import Dataset, Features, ClassLabel
import tempfile, os

with tempfile.TemporaryDirectory() as d:
p = os.path.join(d, "data.parquet")
pq.write_table(pa.table({"label": pa.array([5, 1], type=pa.int64())}), p)
ds = Dataset.from_parquet(p, features=Features({"label": ClassLabel(names=["neg", "pos", "oth"])}))
print(ds["label"]) # [5, 1]
ds.features["label"].int2str(ds["label"][0]) # ValueError: Invalid integer class label 5, raised far from the load site
```

`Dataset.from_dict` with the same `features=` argument raises immediately instead:

```python
Dataset.from_dict({"label": [5, 1]}, features=Features({"label": ClassLabel(names=["neg", "pos", "oth"])}))
# ValueError: Class label 5 greater than configured num_classes 3
```

So a `.jsonl`/`.parquet` file with a bad label and an in-memory dict with the same bad label behave differently: one raises at load time, the other loads silently and only fails (or worse, doesn't fail) downstream.

### Cause

`table_cast` in `src/datasets/table.py` only runs the feature-level cast (`cast_table_to_schema`, which calls `ClassLabel.cast_storage` and does the range check) when the Arrow schema itself changes. `pa.Schema.__eq__` ignores metadata, so when the file's native dtype (`int64` for JSON/Parquet integers) already equals the `ClassLabel`'s storage dtype, only the schema *metadata* differs and `table_cast` takes the cheap `table.replace_schema_metadata(schema.metadata)` branch, never calling `cast_array_to_feature`. `from_dict`/`from_pylist` don't go through `table_cast` for this — they encode through the feature directly, which is why they still validate.

This is the same root cause as #8494 (`cast_column` hitting the identical branch), which #8518 fixes — but #8518 scopes the fix to `Dataset.cast`/`cast_column` only (`table_cast(..., validate_features=True)`), and loaders keep calling `table_cast` with the default `validate_features=False`. That's a deliberate, discussed tradeoff (see the review thread on #8518: a full-validation `table_cast` call costs ~2.26ms vs ~0.006ms for the metadata-only path on a 50k-row/20-column batch, and every packaged loader's `_cast_table` sits on this path), not an oversight — but it leaves the load-time path, which is the more common way users attach a `ClassLabel` to raw integer data, silently unvalidated. I'm filing it separately from #8494 since that issue and #8518's fix are scoped to `cast`/`cast_column` and don't mention the loader path.

Confirmed CSV is not affected: CSV values parse as strings first, so the target `int64` schema never equals the table's own schema and the full cast path always runs.

### Expected behavior

Loading a file with `features=` containing a `ClassLabel` should raise the same `ValueError` as `Dataset.from_dict` when a label index is `>= num_classes`, or at least not silently return an inconsistent dataset.

### Environment

`datasets` 5.0.2.dev0 installed from source at `4e56ddf20a63248a26e14ac848279246f5a35dac`, Python 3.13.12, pyarrow 25.0.1, macOS arm64.

Contributor guide

Open the contributing guide

Research direction

Start in src/datasets/table.py at table_cast, then trace cast_table_to_schema, cast_array_to_feature, and the loaders' _cast_table path. Compare the metadata-only branch with the validation path and review #8518's discussion and benchmarks. Done means file loaders using features= reject out-of-range ClassLabel indices consistently with Dataset.from_dict without an unjustified performance regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.