[Python] Conversion of numpy array of bytes (dtype='S') ignores length
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When converting a numpy array of bytes (dtype='S3' for example) pyarrow seems to make all values the same fixed-length and pad with null characters:
```
In [2]: pa.__version__
Out[2]: '12.0.1'
In [3]: a = np.array([b'a', b'ab', b'abc'])
In [4]: a
Out[4]: array([b'a', b'ab', b'abc'], dtype='|S3')
In [5]: b = pa.array(a, type=pa.string())
In [6]: b
Out[6]:
[
"a",
"ab",
"abc"
]
In [7]: [x.as_py() for x in b]
Out[7]: ['a\x00\x00', 'ab\x00', 'abc']
```
I don't think this is the intended behavior here?
Note that the stringification of `StringArray` hides this issue since it truncates after encountering the first null character. To be observe it you need to convert to Python strings.
This also means that the output of `to_numpy` is not the same as the original array:
```
In [11]: b.to_numpy(zero_copy_only=False)
Out[11]: array(['a\x00\x00', 'ab\x00', 'abc'], dtype=object)
```
If you agree that this is a bug I am happy to contribute a patch.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.