[Python] Convert to_pylist fast paths per range instead of per element
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Follow-up of #50326 (mechanism introduced in GH-50327). The scalar-free conversion currently makes one `_getitem_py` virtual call per element. Several per-element overheads could be batched per range, as suggested in the GH-50327 review:
1. Replace the per-element entry point with a `cdef list _getitem_range_py(self, int64_t offset, int64_t length)` used by `to_pylist` and the list/struct/map container loops, cutting per-element function-call and prologue overhead.
2. Hoist the numeric type-id dispatch out of the loop: `NumericArray._getitem_py` currently walks an if/elif ladder per element even though the type is invariant; a per-range implementation (or per-concrete-class overrides) makes the loop monomorphic.
3. Skip the per-element `IsNull` check when `null_count() == 0`.
4. Read the offsets buffer pointer once per range in the list/map paths instead of calling `value_offset()` twice per row.
None of these change semantics; they compound on top of the structural win in GH-50327 (no per-element Scalar / per-row Array wrapper allocation).
### Component(s)
Python
Contributor guide
Research direction
Start by reading the to_pylist path and the _getitem_py entry points described here, then review follow-up issue #50326 and mechanism GH-50327. Trace the list, struct, map, and NumericArray loops; done means the per-range paths preserve existing semantics while batching the listed overheads and avoiding per-element dispatch where possible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100