apache / apache/arrow

[Python] FixedShapeTensorArray.to_numpy_ndarray() fails with numpy arrays of type string in Pyarrow-16, but works with v15

Open
#43,614 7 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

FixedShapeTensorArray.to_numpy_ndarray() fails with numpy arrays of type string

with pyarrow 16, whereas pyarrow 15 works. As a result, we are unable to load our feather files that were created with older pyarrow versions.
While we should not have stored those strings this way, we have now a significant library of files that contain such constructs.
Code to load the file:
```
from pathlib import Path
from typing import Any

import pyarrow as pa

def _extract_element_feather(column: Any):
import pyarrow as pa

if type(column) == pa.lib.FixedShapeTensorArray:
return column.to_numpy_ndarray()
else:
return column.as_py()

with pa.ipc.open_file(Path("out.feather")) as reader:
table = reader.get_batch(0)
features = {
k.name: _extract_element_feather(table[k.name])
for k in table.schema
}

print(features)
```
Test file: [out.feather.zip](https://github.com/user-attachments/files/16540720/out.feather.zip)

Pyarrow 16:
```
File "/Users/bschindler/Library/Application Support/JetBrains/PyCharm2024.1/scratches/scratch_41.py", line 17, in
features = {
File "/Users/bschindler/Library/Application Support/JetBrains/PyCharm2024.1/scratches/scratch_41.py", line 18, in
k.name: _extract_element_feather(table[k.na
me])
File "/Users/bschindler/Library/Application Support/JetBrains/PyCharm2024.1/scratches/scratch_41.py", line 11, in _extract_element_feather
return column.to_numpy_ndarray()

File "pyarrow/array.pxi", line 4240, in pyarrow.lib.FixedShapeTensorArray.to_numpy_ndarray
File "pyarrow/array.pxi", line 4265, in pyarrow.lib.FixedShapeTensorArray.to_tensor
File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: string is not valid data type for a tensor
```
Pyarrow 15:
```
{'data': array([['Hello World']], dtype=object)}
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.