huggingface / huggingface/datasets

`PandasArrayExtensionDtype._metadata` should be a tuple, not a string

Open Beginner friendly
#8,375 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

There's a small typo in `PandasArrayExtensionDtype` (`src/datasets/features/features.py`):

```python
_metadata = "value_type"
```

Pandas expects `_metadata` to be a tuple of attribute names, like `("value_type",)`, not a plain string ([API documentation ref](https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.ExtensionDtype.html)). Because it's a string, pandas iterates over each character in `"value_type"` when comparing dtypes. So it tries to read attributes named `v`, `a`, `l`, `u`, `e`, etc, and crashes on the first one:

```
AttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v'
```

This shows up whenever:

1. Write a parquet file with array columns (e.g. `array[float64]` from Array2D/Array3D features)
2. Read it back with `pd.read_parquet()` after `import datasets`
3. Filter rows with a boolean mask like `df[mask]`

I ran into this while splitting a LeRobot dataset, but the bug itself is in `datasets`. I think the same root cause is discussed in an issue in LeRobot repo: https://github.com/huggingface/lerobot/issues/2445.

### Steps to reproduce the bug

You can reproduce the error using the code below:
```
import tempfile
from pathlib import Path

import datasets
import pandas as pd
from datasets import Features, Array2D

path = Path(tempfile.mkdtemp()) / "test.parquet"

data = {
"episode_index": [0, 0, 1, 1],
"values": [
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0]],
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0]],
],
}
features = Features({
"episode_index": datasets.Value("int64"),
"values": Array2D(shape=(2, 3), dtype="float64"),
})

ds = datasets.Dataset.from_dict(data, features=features)
ds.to_parquet(path)

df2 = pd.read_parquet(path)
print(df2["values"].dtype)

mask = df2["episode_index"].isin([0])
df2[mask]
```
output:
```
AttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v'
```

### Expected behavior

`df[mask]` should work on DataFrames with `array[float64]` columns read from parquet

### Environment info

- `datasets` version: 5.0.2.dev0
- Platform: Linux-6.8.0-136-generic-x86_64-with-glibc2.39
- Python version: 3.12.13
- `huggingface_hub` version: 1.26.0
- PyArrow version: 25.0.0
- Pandas version: 3.0.5
- `fsspec` version: 2026.6.0

Contributor guide

Open the contributing guide

Research direction

The typo is in src/datasets/features/features.py, in PandasArrayExtensionDtype._metadata. Start by inspecting that declaration and run the supplied parquet reproduction with pandas. Done means df2[mask] works for the array column without raising the reported AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.