apache / apache/arrow

[Python] Convert to_pylist fast paths per range instead of per element

Open
#50,448 2 comments 0 reactions 0 assignees View on GitHub
Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Follow-up of #50326 (mechanism introduced in GH-50327). The scalar-free conversion currently makes one `_getitem_py` virtual call per element. Several per-element overheads could be batched per range, as suggested in the GH-50327 review:

1. Replace the per-element entry point with a `cdef list _getitem_range_py(self, int64_t offset, int64_t length)` used by `to_pylist` and the list/struct/map container loops, cutting per-element function-call and prologue overhead.
2. Hoist the numeric type-id dispatch out of the loop: `NumericArray._getitem_py` currently walks an if/elif ladder per element even though the type is invariant; a per-range implementation (or per-concrete-class overrides) makes the loop monomorphic.
3. Skip the per-element `IsNull` check when `null_count() == 0`.
4. Read the offsets buffer pointer once per range in the list/map paths instead of calling `value_offset()` twice per row.

None of these change semantics; they compound on top of the structural win in GH-50327 (no per-element Scalar / per-row Array wrapper allocation).

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by reading the to_pylist path and the _getitem_py entry points described here, then review follow-up issue #50326 and mechanism GH-50327. Trace the list, struct, map, and NumericArray loops; done means the per-range paths preserve existing semantics while batching the listed overheads and avoiding per-element dispatch where possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.