apache / apache/arrow

[Python] List<Extension> arrays aren't supported in to_pandas calls

Open
#32,791 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

EXTENSION is not in the list of types allowed. I think in order to enable EXTENSION we need to be able to call to_pylist or similar on the original extension array from C++ code, in case there were user provided overrides. Off the top of my head one way of doing this would be to pass through an additional std::unorderd_map where PyObject is the bound to_pylist python function. Are there other alternative that might be cleaner?

**Reporter**: [Micah Kornfield](https://issues.apache.org/jira/browse/ARROW-17535) / @emkornfield
#### Related issues:
- [[Python] Use ExtensionScalar.as_py() as fallback in ExtensionArray to_pandas?](https://github.com/apache/arrow/issues/33134) (relates to)
- [[Python] Nested ExtensionArray conversion to/from pandas/numpy](https://github.com/apache/arrow/issues/33036) (is related to)

**Note**: *This issue was originally created as [ARROW-17535](https://issues.apache.org/jira/browse/ARROW-17535). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Python to_pandas path and the C++ handling of allowed types, then review ExtensionArray.to_pylist and the related ExtensionScalar.as_py fallback issue. Done means List arrays convert through to_pandas while preserving user-provided overrides; resolve the approach alongside the related nested ExtensionArray conversion issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.