huggingface / huggingface/datasets

module 'pyarrow' has no attribute 'PyExtensionType'

Open
#7,742 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

When importing certain libraries, users will encounter the following error which can be traced back to the datasets library.

module 'pyarrow' has no attribute 'PyExtensionType'.

Example issue: https://github.com/explodinggradients/ragas/issues/2170

The issue occurs due to the following. I will proceed to submit a PR with the below fix:

**Issue Reason**
The issue is that PyArrow version 21.0.0 doesn’t have PyExtensionType. This was changed in newer versions of PyArrow. The

PyExtensionType class was renamed to ExtensionType in PyArrow 13.0.0 and later versions.

** Issue Solution**
Making the following changes to the following lib files should temporarily resolve the issue.

I will submit a PR to the dataets library in the meantime.

env_name/lib/python3.10/site-packages/datasets/features/features.py:

```
> 521 self.shape = tuple(shape)
522 self.value_type = dtype
523 self.storage_dtype = self._generate_dtype(self.value_type)
524 - pa.PyExtensionType.__init__(self, self.storage_dtype)
524 + pa.ExtensionType.__init__(self, self.storage_dtype)
525
526 def __reduce__(self):
527 return self.__class__, (
```

Updated venv_name/lib/python3.10/site-packages/datasets/features/features.py:
```
510 _type: str = field(default=“Array5D”, init=False, repr=False)
511
512
513 - class _ArrayXDExtensionType(pa.PyExtensionType):
513 + class _ArrayXDExtensionType(pa.ExtensionType):
514 ndims: Optional[int] = None
515
516 def __init__(self, shape: tuple, dtype: str):
```

### Steps to reproduce the bug

Ragas version: 0.3.1
Python version: 3.11

**Code to Reproduce**

_**In notebook:**_

!pip install ragas
from ragas import evaluate

### Expected behavior

The required package installs without issue.

### Environment info

In Jupyter Notebook.
venv

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.