huggingface / huggingface/datasets

PandasArrayExtensionDtype has no construct_from_string, so comparing the dtype to any string raises AssertionError

Open
#8,467 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

`PandasArrayExtensionDtype` does not implement `construct_from_string`, so it inherits the base `ExtensionDtype` version, which opens with:

```python
@classmethod
def construct_from_string(cls, string):
assert isinstance(cls.name, str), (cls, type(cls.name))
```

`name` is an instance `@property` here (it depends on `value_type`, e.g. `array[float64]`), so `cls.name` is a `property` object and the assertion fires:

```
AssertionError: (, )
```

Pandas calls `construct_from_string` from `ExtensionDtype.__eq__` for **every** comparison of a dtype against a plain string, and it compares dtypes against strings all over its internals (`is_string_dtype`, `astype_is_view`, ...). So any `Array2D`/`Array3D`/`Array4D`/`Array5D` column that reaches pandas breaks a range of ordinary operations with a bare `AssertionError`.

### Steps or code to reproduce the bug

```python
import datasets
from datasets import Array2D, Dataset, Features
from pandas.api.types import is_string_dtype

features = Features({"foo": Array2D(dtype="float64", shape=(2, 2))})
ds = Dataset.from_dict({"foo": [[[1.0, 2.0], [3.0, 4.0]]]}, features=features)
df = ds._data.to_pandas()

df.foo.dtype == "string" # AssertionError
is_string_dtype(df.foo.dtype) # AssertionError
df.astype(object) # AssertionError
df.foo.astype(object) # AssertionError
```

Measured on `main` (`48b7ee7`):

```
FAIL | dtype == 'string' | AssertionError: (PandasArrayExtensionDtype, )
FAIL | dtype == 'array[float64]' (its own name) | AssertionError: (PandasArrayExtensionDtype, )
FAIL | is_string_dtype(dtype) | AssertionError: (PandasArrayExtensionDtype, )
FAIL | df.astype(object) | AssertionError: (PandasArrayExtensionDtype, )
FAIL | df.foo.astype(object) | AssertionError: (PandasArrayExtensionDtype, )
FAIL | df.replace(1.0, 2.0) | AssertionError: (PandasArrayExtensionDtype, )
```

The `astype(object)` traceback bottoms out in pandas at `astype_is_view` -> `is_string_dtype` -> `dtype == "string"` -> `construct_from_string`.

### Expected behavior

Comparing the dtype to an unrelated string should be `False`, not an error, and `df.astype(object)` should work. Per the pandas [extension dtype docs](https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.ExtensionDtype.construct_from_string.html), a subclass whose `name` is not a class-level string has to provide its own `construct_from_string`, raising `TypeError` when the string is not one of its own.

### Relation to #8375

Same root cause family — the pandas `ExtensionDtype` contract is only partly implemented — but a distinct defect with a distinct fix, and the two are independent: #8375 is `_metadata` being a string instead of a tuple.

They do interact. With only `construct_from_string` implemented, pandas gets further and then trips the `_metadata` bug instead:

```
ok | dtype == 'string' | False
FAIL | dtype == 'array[float64]' | AttributeError: ... has no attribute 'v'
FAIL | df.convert_dtypes() | AttributeError: ... has no attribute 'v'
FAIL | df.melt() | AttributeError: ... has no attribute 'v'
FAIL | pd.concat([df, df.astype({'n':'float64'})]) | AttributeError: ... has no attribute 'v'
```

With both applied, all of the above pass. So they are worth landing together even though each is independently correct.

PR: #8468.

### Environment info

- `datasets` version: 5.0.2.dev0 (`main` @ 48b7ee7)
- Python version: 3.11.9
- Platform: Windows 11
- PyArrow version: 25.0.1
- Pandas version: 3.0.5
- NumPy version: 2.4.6

Contributor guide

Open the contributing guide

Research direction

Start at PandasArrayExtensionDtype and inspect how pandas calls construct_from_string during dtype comparisons. Run the provided Array2D reproduction and check comparisons with unrelated and matching strings, is_string_dtype, and astype operations. Done means unrelated strings return False and the listed pandas operations no longer raise AssertionError.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.