[Python] Python extension types aren't usable in struct arrays
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I have a custom PyExtensionType. I would like to use that value for a field of an array, and use that array inside a larger data structure.
I'm able to use the extension type directly in `pa.array`, but if I use a `pa.struct` to wrap it up, it fails.
Here is a minimal reproducer. Running `make_array_ok()` does not error. Running `make_struct_array_not_ok()` results in an error.
```python
import pyarrow as pa
class CustomExtensionType(pa.PyExtensionType):
def __init__(self):
pa.PyExtensionType.__init__(self, pa.int64())
def __reduce__(self):
return CustomExtensionType, ()
def make_array_ok():
data = [1, 2, 3, 4, 5]
typ = CustomExtensionType()
return pa.array(data, typ)
def make_struct_array_not_ok():
data = [{"val": 1}, {"val": 2}, {"val": 3}, {"val": 4}, {"val": 5}]
typ = CustomExtensionType()
struct_type = pa.struct([("val", typ)])
return pa.array(data, type=struct_type)
```
The error is not very clear to me:
```
Traceback (most recent call last):
File "", line 1, in
File "/redacted/struct_demo.py", line 22, in make_struct_array_not_ok
return pa.array(data, type=struct_typ)
File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: extension
```
### Version info
- Pyarrow version 11.0.0
- Python 3.10.10
- OSX, M1 chip (ARM64)
- Installed with wheel (`cp310-cp310-macosx_11_0_arm64`)
### Component(s)
Python
Contributor guide
Research direction
Start with the provided pa.array reproducer, comparing direct PyExtensionType construction with the pa.struct field case, and trace the Python array-conversion entry point it exercises. Done means a struct array containing the custom extension type can be constructed without the ArrowNotImplementedError, with coverage for the reproducer case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100