apache / apache/arrow

[Python] Define whether user-overridden to_pylist on Array subclasses is honored for nested children

Open
#50,604 1 comment 1 reaction 0 assignees View on GitHub
Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

In the review of #50327 it was noted (https://github.com/apache/arrow/pull/50327#discussion_r3629728508) that the bulk conversion does not consult a user-defined `to_pylist` override on `ExtensionArray` subclasses for *nested* children.

This was already inconsistent before #50327:
- `ListScalar.as_py` called `child_values.to_pylist()`, so an override on a list child's class was honored;
- `StructScalar.as_py` converted fields via per-element Scalars, so an override on a struct child's class was ignored.

Top-level overrides dispatch normally through Python method resolution in both the old and the new code, and `ExtensionScalar.as_py` / `__arrow_ext_scalar_class__` — the documented customization point — is honored on all paths.

This issue is to decide the contract:
1. document that nested `to_pylist` overrides are not consulted (only `ExtensionScalar.as_py` is), or
2. detect custom `to_pylist` overrides and route nested conversion through them (sketch in the linked discussion).

Contributor guide

Open the contributing guide

Research direction

Start with the linked discussion in #50327 and compare the existing ListScalar.as_py and StructScalar.as_py paths. Decide whether nested to_pylist overrides are part of the contract or whether only ExtensionScalar.as_py is supported; done means the chosen behavior is consistently implemented and documented, with coverage for nested children.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.