huggingface / huggingface/datasets
PandasArrayExtensionDtype has no construct_from_string, so comparing the dtype to any string raises AssertionError
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
`PandasArrayExtensionDtype` does not implement `construct_from_string`, so it inherits the base `ExtensionDtype` version, which opens with:
```python
@classmethod
def construct_from_string(cls, string):
assert isinstance(cls.name, str), (cls, type(cls.name))
```
`name` is an instance `@property` here (it depends on `value_type`, e.g. `array[float64]`), so `cls.name` is a `property` object and the assertion fires:
```
AssertionError: (, )
```
Pandas calls `construct_from_string` from `ExtensionDtype.__eq__` for **every** comparison of a dtype against a plain string, and it compares dtypes against strings all over its internals (`is_string_dtype`, `astype_is_view`, ...). So any `Array2D`/`Array3D`/`Array4D`/`Array5D` column that reaches pandas breaks a range of ordinary operations with a bare `AssertionError`.
### Steps or code to reproduce the bug
```python
import datasets
from datasets import Array2D, Dataset, Features
from pandas.api.types import is_string_dtype
features = Features({"foo": Array2D(dtype="float64", shape=(2, 2))})
ds = Dataset.from_dict({"foo": [[[1.0, 2.0], [3.0, 4.0]]]}, features=features)
df = ds._data.to_pandas()
df.foo.dtype == "string" # AssertionError
is_string_dtype(df.foo.dtype) # AssertionError
df.astype(object) # AssertionError
df.foo.astype(object) # AssertionError
```
Measured on `main` (`48b7ee7`):
```
FAIL | dtype == 'string' | AssertionError: (PandasArrayExtensionDtype, )
FAIL | dtype == 'array[float64]' (its own name) | AssertionError: (PandasArrayExtensionDtype, )
FAIL | is_string_dtype(dtype) | AssertionError: (PandasArrayExtensionDtype, )
FAIL | df.astype(object) | AssertionError: (PandasArrayExtensionDtype, )
FAIL | df.foo.astype(object) | AssertionError: (PandasArrayExtensionDtype, )
FAIL | df.replace(1.0, 2.0) | AssertionError: (PandasArrayExtensionDtype, )
```
The `astype(object)` traceback bottoms out in pandas at `astype_is_view` -> `is_string_dtype` -> `dtype == "string"` -> `construct_from_string`.
### Expected behavior
Comparing the dtype to an unrelated string should be `False`, not an error, and `df.astype(object)` should work. Per the pandas [extension dtype docs](https://pandas.pydata.org/docs/reference/api/pandas.api.extensions.ExtensionDtype.construct_from_string.html), a subclass whose `name` is not a class-level string has to provide its own `construct_from_string`, raising `TypeError` when the string is not one of its own.
### Relation to #8375
Same root cause family — the pandas `ExtensionDtype` contract is only partly implemented — but a distinct defect with a distinct fix, and the two are independent: #8375 is `_metadata` being a string instead of a tuple.
They do interact. With only `construct_from_string` implemented, pandas gets further and then trips the `_metadata` bug instead:
```
ok | dtype == 'string' | False
FAIL | dtype == 'array[float64]' | AttributeError: ... has no attribute 'v'
FAIL | df.convert_dtypes() | AttributeError: ... has no attribute 'v'
FAIL | df.melt() | AttributeError: ... has no attribute 'v'
FAIL | pd.concat([df, df.astype({'n':'float64'})]) | AttributeError: ... has no attribute 'v'
```
With both applied, all of the above pass. So they are worth landing together even though each is independently correct.
PR: #8468.
### Environment info
- `datasets` version: 5.0.2.dev0 (`main` @ 48b7ee7)
- Python version: 3.11.9
- Platform: Windows 11
- PyArrow version: 25.0.1
- Pandas version: 3.0.5
- NumPy version: 2.4.6
Contributor guide
Research direction
Start at PandasArrayExtensionDtype and inspect how pandas calls construct_from_string during dtype comparisons. Run the provided Array2D reproduction and check comparisons with unrelated and matching strings, is_string_dtype, and astype operations. Done means unrelated strings return False and the listed pandas operations no longer raise AssertionError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100