apache / apache/arrow

[Python] Conversion to/from numpy 2.0+ new StringDType

Open
#42,018 13 comments 0 reactions 1 assignee Claimed by @alippai View on GitHub
Component: Python Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

With the upcoming numpy 2.0 release, they introduce a new variable-length string dtype, [StringDType](https://numpy.org/devdocs/reference/routines.dtypes.html#numpy.dtypes.StringDType) (https://numpy.org/devdocs/release/2.0.0-notes.html#highlights).

Currently trying to convert that obviously gives an error:

```python
>>> arr = np.array(["some", "strings"], dtype=np.dtypes.StringDType())
>>> arr
array(['some', 'strings'], dtype=StringDType())
>>> pa.array(arr)
...
ArrowNotImplementedError: Unsupported numpy type 2056
```

But ideally we should support this dtype both for conversion from numpy -> arrow, as in the arrow -> numpy conversion (although here it should probably be opt-in and by default still return object dtype)

Numpy provides a C API to access the individual string elements: https://numpy.org/devdocs/reference/c-api/strings.html

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.