apache / apache/arrow

[Python] Cannot use sparse_union/dense_union types with pa.Table.from_pylist()

Open
#44,182 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

An exception is raised when attempting to use a sparse or dense union with `pa.Table.from_pylist()`.

**Steps to Reproduce**:
Run the following Python code:

```python
import pyarrow as pa

schema = pa.schema([
pa.field("data",
pa.sparse_union([
pa.field("age", pa.int32()),
pa.field("weight", pa.int32())
])
)
])

data = [
{"data": {"age": 20}},
]

table = pa.Table.from_pylist(data, schema=schema)
```

**Expected Behavior:**
The code should create a `pa.Table` using the provided schema and data.

**Actual Behavior**
The following exception is raised:

```
pyarrow.lib.ArrowNotImplementedError: sparse_union
```

With a traceback of:

```
Traceback (most recent call last):
File "/Users/rusty/Development/flight-cloud/src/flight_cloud/example-union.py", line 7, in
table = pa.Table.from_pylist(data, schema=schema)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 1984, in pyarrow.lib._Tabular.from_pylist
File "pyarrow/table.pxi", line 6230, in pyarrow.lib._from_pylist
File "pyarrow/table.pxi", line 4758, in pyarrow.lib.Table.from_arrays
File "pyarrow/table.pxi", line 1562, in pyarrow.lib._sanitize_arrays
File "pyarrow/array.pxi", line 398, in pyarrow.lib.asarray
File "pyarrow/array.pxi", line 368, in pyarrow.lib.array
File "pyarrow/array.pxi", line 42, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: sparse_union
```

This occurs both when using `pa.sparse_union` and `pa.dense_union`.

**PyArrow version: 18.0.0.dev**

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.