apache / apache/arrow

[Python] Conversion of numpy array of bytes (dtype='S') ignores length

Open
#36,308 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When converting a numpy array of bytes (dtype='S3' for example) pyarrow seems to make all values the same fixed-length and pad with null characters:
```
In [2]: pa.__version__
Out[2]: '12.0.1'

In [3]: a = np.array([b'a', b'ab', b'abc'])

In [4]: a
Out[4]: array([b'a', b'ab', b'abc'], dtype='|S3')

In [5]: b = pa.array(a, type=pa.string())

In [6]: b
Out[6]:

[
"a",
"ab",
"abc"
]

In [7]: [x.as_py() for x in b]
Out[7]: ['a\x00\x00', 'ab\x00', 'abc']
```
I don't think this is the intended behavior here?

Note that the stringification of `StringArray` hides this issue since it truncates after encountering the first null character. To be observe it you need to convert to Python strings.

This also means that the output of `to_numpy` is not the same as the original array:
```
In [11]: b.to_numpy(zero_copy_only=False)
Out[11]: array(['a\x00\x00', 'ab\x00', 'abc'], dtype=object)
```

If you agree that this is a bug I am happy to contribute a patch.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.