huggingface / huggingface/datasets
load_dataset silently accepts out-of-range ClassLabel indices when the source dtype already matches
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
Loading a file whose parsed Arrow dtype already matches a declared `ClassLabel`'s storage type (`int64`) skips the range check entirely. Out-of-range indices load with no error and no warning, and only surface later at `int2str` time or as silently wrong labels during training.
```python
from datasets import Dataset, Features, ClassLabel
import json, tempfile, os
with tempfile.TemporaryDirectory() as d:
p = os.path.join(d, "data.jsonl")
with open(p, "w") as f:
f.write(json.dumps({"label": 5}) + "\n")
f.write(json.dumps({"label": 1}) + "\n")
ds = Dataset.from_json(p, features=Features({"label": ClassLabel(names=["neg", "pos", "oth"])}))
print(ds["label"]) # [5, 1], no error, only 3 classes declared
```
Same result with Parquet:
```python
import pyarrow as pa, pyarrow.parquet as pq
from datasets import Dataset, Features, ClassLabel
import tempfile, os
with tempfile.TemporaryDirectory() as d:
p = os.path.join(d, "data.parquet")
pq.write_table(pa.table({"label": pa.array([5, 1], type=pa.int64())}), p)
ds = Dataset.from_parquet(p, features=Features({"label": ClassLabel(names=["neg", "pos", "oth"])}))
print(ds["label"]) # [5, 1]
ds.features["label"].int2str(ds["label"][0]) # ValueError: Invalid integer class label 5, raised far from the load site
```
`Dataset.from_dict` with the same `features=` argument raises immediately instead:
```python
Dataset.from_dict({"label": [5, 1]}, features=Features({"label": ClassLabel(names=["neg", "pos", "oth"])}))
# ValueError: Class label 5 greater than configured num_classes 3
```
So a `.jsonl`/`.parquet` file with a bad label and an in-memory dict with the same bad label behave differently: one raises at load time, the other loads silently and only fails (or worse, doesn't fail) downstream.
### Cause
`table_cast` in `src/datasets/table.py` only runs the feature-level cast (`cast_table_to_schema`, which calls `ClassLabel.cast_storage` and does the range check) when the Arrow schema itself changes. `pa.Schema.__eq__` ignores metadata, so when the file's native dtype (`int64` for JSON/Parquet integers) already equals the `ClassLabel`'s storage dtype, only the schema *metadata* differs and `table_cast` takes the cheap `table.replace_schema_metadata(schema.metadata)` branch, never calling `cast_array_to_feature`. `from_dict`/`from_pylist` don't go through `table_cast` for this — they encode through the feature directly, which is why they still validate.
This is the same root cause as #8494 (`cast_column` hitting the identical branch), which #8518 fixes — but #8518 scopes the fix to `Dataset.cast`/`cast_column` only (`table_cast(..., validate_features=True)`), and loaders keep calling `table_cast` with the default `validate_features=False`. That's a deliberate, discussed tradeoff (see the review thread on #8518: a full-validation `table_cast` call costs ~2.26ms vs ~0.006ms for the metadata-only path on a 50k-row/20-column batch, and every packaged loader's `_cast_table` sits on this path), not an oversight — but it leaves the load-time path, which is the more common way users attach a `ClassLabel` to raw integer data, silently unvalidated. I'm filing it separately from #8494 since that issue and #8518's fix are scoped to `cast`/`cast_column` and don't mention the loader path.
Confirmed CSV is not affected: CSV values parse as strings first, so the target `int64` schema never equals the table's own schema and the full cast path always runs.
### Expected behavior
Loading a file with `features=` containing a `ClassLabel` should raise the same `ValueError` as `Dataset.from_dict` when a label index is `>= num_classes`, or at least not silently return an inconsistent dataset.
### Environment
`datasets` 5.0.2.dev0 installed from source at `4e56ddf20a63248a26e14ac848279246f5a35dac`, Python 3.13.12, pyarrow 25.0.1, macOS arm64.
Contributor guide
Research direction
Start in src/datasets/table.py at table_cast, then trace cast_table_to_schema, cast_array_to_feature, and the loaders' _cast_table path. Compare the metadata-only branch with the validation path and review #8518's discussion and benchmarks. Done means file loaders using features= reject out-of-range ClassLabel indices consistently with Dataset.from_dict without an unjustified performance regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100