huggingface / huggingface/datasets

ArrayXD silently reshapes inputs with incompatible nested dimensions

Open
#8,507 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

`Array2D` and the other `ArrayXD` features accept nested input whose dimensions do not match the declared shape when the total number of values happens to match.

For example, data shaped like `(2, 2)` can be created from rows of lengths `3` and `1`. When the value is read back, the rows are silently regrouped into lengths `2` and `2`, so the returned structure is different from the input.

### Steps to reproduce the bug

from datasets import Array2D, Dataset, Features

data = {"col": [[[1, 2, 3], [4]]]}
features = Features({"col": Array2D(shape=(2, 2), dtype="int64")})

dataset = Dataset.from_dict(data, features=features)

print(data["col"][0])
print(dataset[0]["col"])
```
Output:
[[1, 2, 3], [4]]
[[1, 2], [3, 4]]
```

### Expected behavior

Dataset creation should reject values whose fixed dimensions do not match the declared `ArrayXD` shape. It should not silently move values between nested rows.

I can work on a regression test and a validation fix if this behavior should be rejected during Arrow conversion.

### Environment info

- `datasets` version: 5.0.2.dev0 (`main` at `836b82e0544`)
- Platform: macOS-26.5.2-arm64-arm-64bit
- Python version: 3.12.13
- `huggingface_hub` version: 1.28.0
- PyArrow version: 25.0.1
- Pandas version: 3.0.5
- `fsspec` version: 2026.6.0

Contributor guide

Open the contributing guide

Research direction

Start from Array2D and the other ArrayXD feature paths exercised by Dataset.from_dict, then trace their Arrow conversion and current dimension handling. Add a regression test using incompatible nested row lengths with a matching total value count, and make sure dataset creation rejects the input instead of reshaping it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.