[Python] Define whether user-overridden to_pylist on Array subclasses is honored for nested children
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
In the review of #50327 it was noted (https://github.com/apache/arrow/pull/50327#discussion_r3629728508) that the bulk conversion does not consult a user-defined `to_pylist` override on `ExtensionArray` subclasses for *nested* children.
This was already inconsistent before #50327:
- `ListScalar.as_py` called `child_values.to_pylist()`, so an override on a list child's class was honored;
- `StructScalar.as_py` converted fields via per-element Scalars, so an override on a struct child's class was ignored.
Top-level overrides dispatch normally through Python method resolution in both the old and the new code, and `ExtensionScalar.as_py` / `__arrow_ext_scalar_class__` — the documented customization point — is honored on all paths.
This issue is to decide the contract:
1. document that nested `to_pylist` overrides are not consulted (only `ExtensionScalar.as_py` is), or
2. detect custom `to_pylist` overrides and route nested conversion through them (sketch in the linked discussion).
Contributor guide
Research direction
Start with the linked discussion in #50327 and compare the existing ListScalar.as_py and StructScalar.as_py paths. Decide whether nested to_pylist overrides are part of the contract or whether only ExtensionScalar.as_py is supported; done means the chosen behavior is consistently implemented and documented, with coverage for nested children.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100