[Python] Conversion from `np.ndarray` with bytes data type to binary pyarrow array discards all bytes after the first null byte.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
## Example 1
```
import numpy as np
import pyarrow as pa
np_array = np.array([
b'This data contains \x00 and bytes after it',
b'This data does not have any null bytes.'])
table = pa.Table.from_pydict({'sample': np_array})
print(table['sample'][0].as_py())
```
Expected output `b'This data contains \x00 and bytes after it'`, actual output `b'This data contains '`. Note: `type(table['sample'])` in this example is `pyarrow.lib.ChunkedArray`. `table['sample'][1]` is `b'This data does not have any null bytes.'` as expected. The byte arrays in this example look like human-readable strings (except for the first null byte), but in reality we are trying to encode binary data as a fixed length byte array and store it in a (variable length for compatibility with HuggingFace) byte array parquet column.
## Example 2
With the same `np_array`:
```
pa_array = pa.array(np_array, type=pa.binary())
print(pa_array[0])
```
Expected output `b'This data contains \x00 and bytes after it'`, actual output `b'This data contains '`. Note: `type(pa_array)` is `pyarrow.lib.BinaryArray`.
## Possible workaround
Replacing `table = pa.Table.from_pydict({'sample': np_array})` with `table = pa.Table.from_pydict({'sample': pa.array(np_array, type=pa.binary(np_array.itemsize)).cast(target_type=pa.binary())})` seems to produce the intended result.
Note that in our usecase we want the resulting dtype to be `pa.binary()` (BYTE_ARRAY in parquet), not `pa.binary(35) (FIXED_LEN_BYTE_ARRAY in parquet), because the latter does not seem to be supported by HuggingFace datasets library.
## System info
The above outputs are on Ubuntu 22.04.4 LTS with Python 3.10.11, pyarrow version 12.0.1 (using parquet-cpp-arrow version 12.0.1).
## Related issues
This seems to be similar but distinct from #36308 , where the input contained byte arrays of different lengths and a conversion to `pa.string()` column was desired.
An unexpected parquet file size, which was later traced to this issue, was originally described (by me) in [this StackOverflow post](https://stackoverflow.com/questions/78359858/why-would-a-parquet-file-get-larger-when-fixed-len-byte-array-data-type-is-used/78388003#78388003).
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.