apache / apache/arrow

[C++][Python] Python to Arrow array conversion sometimes ignores "safe=False" or is using unsafe instead of safe cast

Open
#36,202 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

There are two cases where the array constructor taking the python conversion path (`python_to_arrow.cc`) doesn't handle the safe keyword properly.

### The safe keyword is set to True by default and is ignored if passed

- One example where python list is taking `python_to_arrow.cc` code path are Decimals (which are python objects). Here the conversion from Decimal to int does not overflow by default and one can’t turn it off with `safe=False`:

```python
>>> import pyarrow as pa
>>> pa.array([Decimal('1234')]).cast(pa.int8(), safe=False)

[
-46
]
>>> pa.array([Decimal('1234')], pa.int8(), safe=False)
Traceback (most recent call last):
...
ArrowInvalid: Value 1234 too large to fit in C integer type
```

- Another example is JSON data with nested data (list type). The conversion is taking python_to_arrow.cc code path where safe keyword is also ignored. See example in https://github.com/apache/arrow/issues/31402.

### In some cases the safe keyword is ignored and does unsafe conversions

- Nested case

```python
>>> import pyarrow as pa
>>> pa.array(np.array([[1.5], [2.5, 3.5]], dtype=object), type=pa.list_(pa.int64()), safe=True)

[
[
1
],
[
2,
3
]
]
```

- Primitive array case. Numpy array and python list take a different code path (`numpy_to_array.cc` vs `python_to_arrow.cc`):

```python
>>> import pyarrow as pa
>>> pa.array(np.array([1.5, 2.5]), type=pa.int64(), safe=True)
...
ArrowInvalid: Float value 1.5 was truncated converting to int64
```

vs
```python
>>> import pyarrow as pa
>>> pa.array([1.5, 2.5], type=pa.int64(), safe=True)

[
1,
2
]
```

- Another example of wrong handling of safe keyword is when using nested data in pandas (an object thus taking the `python_to_arrow.cc` code path):

```python
>>> import pandas as pd
>>> import pyarrow as pa
>>> int_dataframe = pd.DataFrame({"array": [[1, 2]]})
>>> float_dataframe = pd.DataFrame({"array": [[1.5, 2.3]]})
>>> int_table = pa.Table.from_pandas(int_dataframe)
>>> table = pa.Table.from_pandas(float_dataframe, schema=int_table.schema)
>>> table
pyarrow.Table
array: list
child 0, item: int64
----
array: [[[1,2]]]
```

cc @jorisvandenbossche @danepitkin

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.