[Python] Implement pa.array() with type=union type
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
It would be useful to be able to generate unions during type inference:
```Java
In [11]: pa.array([{'a': 1, 'b': 'string'}, {'b': 2}])
---------------------------------------------------------------------------
ArrowTypeError Traceback (most recent call last)
in ()
----> 1 pa.array([{'a': 1, 'b': 'string'}, {'b': 2}])
~/code/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()
179 if mask is not None:
180 raise ValueError("Masks only supported with ndarray-like inputs")
--> 181 return _sequence_to_array(obj, size, type, pool)
182
183
~/code/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()
24 if size is None:
25 with nogil:
---> 26 check_status(ConvertPySequence(sequence, pool, &out))
27 else:
28 c_size = size
~/code/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
89 raise ArrowNotImplementedError(message)
90 elif status.IsTypeError():
---> 91 raise ArrowTypeError(message)
92 elif status.IsCapacityError():
93 raise ArrowCapacityError(message)
ArrowTypeError: ../src/arrow/python/builtin_convert.cc:794 code: AppendPySequence(seq, size, real_type, builder.get())
../src/arrow/python/iterators.h:60 code: func(value)
../src/arrow/python/builtin_convert.cc:619 code: value_converters_[i]->AppendSingle(valueobj ? valueobj : Py_None)
../src/arrow/python/builtin_convert.cc:414 code: internal::CIntFromPython(obj, &value)
../src/arrow/python/helpers.cc:259 code: CheckPyError()
an integer is required (got type str)
In [12]: pa.array([5, 'str', False])
---------------------------------------------------------------------------
ArrowTypeError Traceback (most recent call last)
in ()
----> 1 pa.array([5, 'str', False])
~/code/arrow/python/pyarrow/array.pxi in pyarrow.lib.array()
179 if mask is not None:
180 raise ValueError("Masks only supported with ndarray-like inputs")
--> 181 return _sequence_to_array(obj, size, type, pool)
182
183
~/code/arrow/python/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()
24 if size is None:
25 with nogil:
---> 26 check_status(ConvertPySequence(sequence, pool, &out))
27 else:
28 c_size = size
~/code/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
89 raise ArrowNotImplementedError(message)
90 elif status.IsTypeError():
---> 91 raise ArrowTypeError(message)
92 elif status.IsCapacityError():
93 raise ArrowCapacityError(message)
ArrowTypeError: ../src/arrow/python/builtin_convert.cc:794 code: AppendPySequence(seq, size, real_type, builder.get())
../src/arrow/python/iterators.h:60 code: func(value)
../src/arrow/python/builtin_convert.cc:414 code: internal::CIntFromPython(obj, &value)
../src/arrow/python/helpers.cc:259 code: CheckPyError()
an integer is required (got type str)
```
**Reporter**: [Wes McKinney](https://issues.apache.org/jira/browse/ARROW-2774) / @wesm
#### Related issues:
- [[Python] Create UnionArray from mixed-type pandas categorical](https://github.com/apache/arrow/issues/19391) (relates to)
**Note**: *This issue was originally created as [ARROW-2774](https://issues.apache.org/jira/browse/ARROW-2774). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start at the pyarrow.lib.array and _sequence_to_array entry points, then follow the ConvertPySequence and AppendPySequence paths shown in the traceback. Establish the expected union inference for the mixed dictionaries and scalar values, and verify that pa.array produces the intended union arrays for both examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100