apache / apache/arrow

[Python] Conversion from `np.ndarray` with bytes data type to binary pyarrow array discards all bytes after the first null byte.

Open
#41,388 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

## Example 1
```
import numpy as np
import pyarrow as pa
np_array = np.array([
b'This data contains \x00 and bytes after it',
b'This data does not have any null bytes.'])
table = pa.Table.from_pydict({'sample': np_array})
print(table['sample'][0].as_py())
```
Expected output `b'This data contains \x00 and bytes after it'`, actual output `b'This data contains '`. Note: `type(table['sample'])` in this example is `pyarrow.lib.ChunkedArray`. `table['sample'][1]` is `b'This data does not have any null bytes.'` as expected. The byte arrays in this example look like human-readable strings (except for the first null byte), but in reality we are trying to encode binary data as a fixed length byte array and store it in a (variable length for compatibility with HuggingFace) byte array parquet column.

## Example 2
With the same `np_array`:
```
pa_array = pa.array(np_array, type=pa.binary())
print(pa_array[0])
```
Expected output `b'This data contains \x00 and bytes after it'`, actual output `b'This data contains '`. Note: `type(pa_array)` is `pyarrow.lib.BinaryArray`.

## Possible workaround
Replacing `table = pa.Table.from_pydict({'sample': np_array})` with `table = pa.Table.from_pydict({'sample': pa.array(np_array, type=pa.binary(np_array.itemsize)).cast(target_type=pa.binary())})` seems to produce the intended result.

Note that in our usecase we want the resulting dtype to be `pa.binary()` (BYTE_ARRAY in parquet), not `pa.binary(35) (FIXED_LEN_BYTE_ARRAY in parquet), because the latter does not seem to be supported by HuggingFace datasets library.

## System info
The above outputs are on Ubuntu 22.04.4 LTS with Python 3.10.11, pyarrow version 12.0.1 (using parquet-cpp-arrow version 12.0.1).

## Related issues
This seems to be similar but distinct from #36308 , where the input contained byte arrays of different lengths and a conversion to `pa.string()` column was desired.

An unexpected parquet file size, which was later traced to this issue, was originally described (by me) in [this StackOverflow post](https://stackoverflow.com/questions/78359858/why-would-a-parquet-file-get-larger-when-fixed-len-byte-array-data-type-is-used/78388003#78388003).

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.