apache / apache/arrow

[Python] Support array indices in pc.list_element

Open
#48,349 3 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

# Description:
pc.list_element(array, index) only accepts scalar integer indices. This prevents efficient vectorized operations. For example, to get the last element from each list:

```python
import pyarrow as pa
import pyarrow.compute as pc
arr = pa.array([[1, 2, 3], [4, 5], [6, 7, 8, 9]])
```
# Desired but not supported:
```python
indices = pa.array([2, 1, 3]) # or [-1, -1, -1]
result = pc.list_element(arr, indices) # Should return [3, 5, 9]
```
# Use Case:
pandas needs this for efficient .list[-1] operations with pyarrow-backed arrays. Currently, pandas must use slow Python loops as a workaround.

# Proposed Solution:
Extend pc.list_element to accept array indices (like other compute functions do). This would enable vectorized element access from lists.

# Related Issues:

pandas issue: (#63221)

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.