huggingface / huggingface/datasets
ArrayXD silently reshapes inputs with incompatible nested dimensions
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
`Array2D` and the other `ArrayXD` features accept nested input whose dimensions do not match the declared shape when the total number of values happens to match.
For example, data shaped like `(2, 2)` can be created from rows of lengths `3` and `1`. When the value is read back, the rows are silently regrouped into lengths `2` and `2`, so the returned structure is different from the input.
### Steps to reproduce the bug
from datasets import Array2D, Dataset, Features
data = {"col": [[[1, 2, 3], [4]]]}
features = Features({"col": Array2D(shape=(2, 2), dtype="int64")})
dataset = Dataset.from_dict(data, features=features)
print(data["col"][0])
print(dataset[0]["col"])
```
Output:
[[1, 2, 3], [4]]
[[1, 2], [3, 4]]
```
### Expected behavior
Dataset creation should reject values whose fixed dimensions do not match the declared `ArrayXD` shape. It should not silently move values between nested rows.
I can work on a regression test and a validation fix if this behavior should be rejected during Arrow conversion.
### Environment info
- `datasets` version: 5.0.2.dev0 (`main` at `836b82e0544`)
- Platform: macOS-26.5.2-arm64-arm-64bit
- Python version: 3.12.13
- `huggingface_hub` version: 1.28.0
- PyArrow version: 25.0.1
- Pandas version: 3.0.5
- `fsspec` version: 2026.6.0
Contributor guide
Research direction
Start from Array2D and the other ArrayXD feature paths exercised by Dataset.from_dict, then trace their Arrow conversion and current dimension handling. Add a regression test using incompatible nested row lengths with a matching total value count, and make sure dataset creation rejects the input instead of reshaping it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100