apache / apache/arrow

[C++][Python] pa.array raises for mixed scalar types (float16 + int)

Open
#25,859 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Priority: Major Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

pa.array raises with mixed scalars for np.float16 and a numpy int: 
```java

In [1]: import numpy as np
...: import pyarrow as pa
...: pa.array([np.float16(1.5), np.int64(5)])
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
in
1 import numpy as np
2 import pyarrow as pa
----> 3 pa.array([np.float16(1.5), np.int64(5)])
~/git_repo/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()
269 else:
270 # ConvertPySequence does strict conversion if type is explicitly passed
--> 271 return _sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
272
273
~/git_repo/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()
38
39 with nogil:
---> 40 check_status(ConvertPySequence(sequence, mask, options, &out))
41
42 if out.get().num_chunks() == 1:
~/git_repo/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
82
83 if status.IsInvalid():
---> 84 raise ArrowInvalid(message)
85 elif status.IsIOError():
86 # Note: OSError constructor is
ArrowInvalid: Cannot mix NumPy dtypes float16 and int64 
```
On master I get the same behaviour with all numpy ints (signed and unsigned)

**Reporter**: [Andrew Wieteska](https://issues.apache.org/jira/browse/ARROW-9817) / @arw2019

**Note**: *This issue was originally created as [ARROW-9817](https://issues.apache.org/jira/browse/ARROW-9817). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the NumPy float16 and int64 values shown in the issue, then start at python/pyarrow/array.pxi, especially pyarrow.lib.array(), _sequence_to_array(), and ConvertPySequence. Done means pa.array accepts the mixed scalar sequence without raising ArrowInvalid while preserving the expected array values.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.