apache / apache/arrow

[Python] Implement pa.array() with type=union type

Open
#19,157 10 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

It would be useful to be able to generate unions during type inference:

```Java

In [11]: pa.array([{'a': 1, 'b': 'string'}, {'b': 2}])
---------------------------------------------------------------------------
ArrowTypeError Traceback (most recent call last)
in ()
----> 1 pa.array([{'a': 1, 'b': 'string'}, {'b': 2}])

~/code/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()
179 if mask is not None:
180 raise ValueError("Masks only supported with ndarray-like inputs")
--> 181 return _sequence_to_array(obj, size, type, pool)
182
183

~/code/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()
24 if size is None:
25 with nogil:
---> 26 check_status(ConvertPySequence(sequence, pool, &out))
27 else:
28 c_size = size

~/code/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
89 raise ArrowNotImplementedError(message)
90 elif status.IsTypeError():
---> 91 raise ArrowTypeError(message)
92 elif status.IsCapacityError():
93 raise ArrowCapacityError(message)

ArrowTypeError: ../src/arrow/python/builtin_convert.cc:794 code: AppendPySequence(seq, size, real_type, builder.get())
../src/arrow/python/iterators.h:60 code: func(value)
../src/arrow/python/builtin_convert.cc:619 code: value_converters_[i]->AppendSingle(valueobj ? valueobj : Py_None)
../src/arrow/python/builtin_convert.cc:414 code: internal::CIntFromPython(obj, &value)
../src/arrow/python/helpers.cc:259 code: CheckPyError()
an integer is required (got type str)

In [12]: pa.array([5, 'str', False])
---------------------------------------------------------------------------
ArrowTypeError Traceback (most recent call last)
in ()
----> 1 pa.array([5, 'str', False])

~/code/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()
179 if mask is not None:
180 raise ValueError("Masks only supported with ndarray-like inputs")
--> 181 return _sequence_to_array(obj, size, type, pool)
182
183

~/code/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()
24 if size is None:
25 with nogil:
---> 26 check_status(ConvertPySequence(sequence, pool, &out))
27 else:
28 c_size = size

~/code/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
89 raise ArrowNotImplementedError(message)
90 elif status.IsTypeError():
---> 91 raise ArrowTypeError(message)
92 elif status.IsCapacityError():
93 raise ArrowCapacityError(message)

ArrowTypeError: ../src/arrow/python/builtin_convert.cc:794 code: AppendPySequence(seq, size, real_type, builder.get())
../src/arrow/python/iterators.h:60 code: func(value)
../src/arrow/python/builtin_convert.cc:414 code: internal::CIntFromPython(obj, &value)
../src/arrow/python/helpers.cc:259 code: CheckPyError()
an integer is required (got type str)
```

**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-2774) / @wesm
#### Related issues:
- [[Python] Create UnionArray from mixed-type pandas categorical](https://github.com/apache/arrow/issues/19391) (relates to)

**Note**: *This issue was originally created as [ARROW-2774](https://issues.apache.org/jira/browse/ARROW-2774). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start at the pyarrow.lib.array and _sequence_to_array entry points, then follow the ConvertPySequence and AppendPySequence paths shown in the traceback. Establish the expected union inference for the mixed dictionaries and scalar values, and verify that pa.array produces the intended union arrays for both examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.