huggingface / huggingface/datasets
`PandasArrayExtensionDtype._metadata` should be a tuple, not a string
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
There's a small typo in `PandasArrayExtensionDtype` (`src/datasets/features/features.py`):
```python
_metadata = "value_type"
```
Pandas expects `_metadata` to be a tuple of attribute names, like `("value_type",)`, not a plain string ([API documentation ref](https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.ExtensionDtype.html)). Because it's a string, pandas iterates over each character in `"value_type"` when comparing dtypes. So it tries to read attributes named `v`, `a`, `l`, `u`, `e`, etc, and crashes on the first one:
```
AttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v'
```
This shows up whenever:
1. Write a parquet file with array columns (e.g. `array[float64]` from Array2D/Array3D features)
2. Read it back with `pd.read_parquet()` after `import datasets`
3. Filter rows with a boolean mask like `df[mask]`
I ran into this while splitting a LeRobot dataset, but the bug itself is in `datasets`. I think the same root cause is discussed in an issue in LeRobot repo: https://github.com/huggingface/lerobot/issues/2445.
### Steps to reproduce the bug
You can reproduce the error using the code below:
```
import tempfile
from pathlib import Path
import datasets
import pandas as pd
from datasets import Features, Array2D
path = Path(tempfile.mkdtemp()) / "test.parquet"
data = {
"episode_index": [0, 0, 1, 1],
"values": [
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0]],
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0]],
],
}
features = Features({
"episode_index": datasets.Value("int64"),
"values": Array2D(shape=(2, 3), dtype="float64"),
})
ds = datasets.Dataset.from_dict(data, features=features)
ds.to_parquet(path)
df2 = pd.read_parquet(path)
print(df2["values"].dtype)
mask = df2["episode_index"].isin([0])
df2[mask]
```
output:
```
AttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v'
```
### Expected behavior
`df[mask]` should work on DataFrames with `array[float64]` columns read from parquet
### Environment info
- `datasets` version: 5.0.2.dev0
- Platform: Linux-6.8.0-136-generic-x86_64-with-glibc2.39
- Python version: 3.12.13
- `huggingface_hub` version: 1.26.0
- PyArrow version: 25.0.0
- Pandas version: 3.0.5
- `fsspec` version: 2026.6.0
Contributor guide
Research direction
The typo is in src/datasets/features/features.py, in PandasArrayExtensionDtype._metadata. Start by inspecting that declaration and run the supplied parquet reproduction with pandas. Done means df2[mask] works for the array column without raising the reported AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100