apache / apache/arrow

[Python] Python extension types aren't usable in struct arrays

Open
#34,985 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I have a custom PyExtensionType. I would like to use that value for a field of an array, and use that array inside a larger data structure.

I'm able to use the extension type directly in `pa.array`, but if I use a `pa.struct` to wrap it up, it fails.

Here is a minimal reproducer. Running `make_array_ok()` does not error. Running `make_struct_array_not_ok()` results in an error.

```python
import pyarrow as pa

class CustomExtensionType(pa.PyExtensionType):
def __init__(self):
pa.PyExtensionType.__init__(self, pa.int64())

def __reduce__(self):
return CustomExtensionType, ()

def make_array_ok():
data = [1, 2, 3, 4, 5]
typ = CustomExtensionType()
return pa.array(data, typ)

def make_struct_array_not_ok():
data = [{"val": 1}, {"val": 2}, {"val": 3}, {"val": 4}, {"val": 5}]
typ = CustomExtensionType()
struct_type = pa.struct([("val", typ)])
return pa.array(data, type=struct_type)
```

The error is not very clear to me:
```
Traceback (most recent call last):
File "", line 1, in
File "/redacted/struct_demo.py", line 22, in make_struct_array_not_ok
return pa.array(data, type=struct_typ)
File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: extension
```
### Version info

- Pyarrow version 11.0.0
- Python 3.10.10
- OSX, M1 chip (ARM64)
- Installed with wheel (`cp310-cp310-macosx_11_0_arm64`)

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the provided pa.array reproducer, comparing direct PyExtensionType construction with the pa.struct field case, and trace the Python array-conversion entry point it exercises. Done means a struct array containing the custom extension type can be constructed without the ArrowNotImplementedError, with coverage for the reproducer case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.