[Python] List<Extension> arrays aren't supported in to_pandas calls
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
EXTENSION is not in the list of types allowed. I think in order to enable EXTENSION we need to be able to call to_pylist or similar on the original extension array from C++ code, in case there were user provided overrides. Off the top of my head one way of doing this would be to pass through an additional std::unorderd_map where PyObject is the bound to_pylist python function. Are there other alternative that might be cleaner?
**Reporter**: [Micah Kornfield](https://issues.apache.org/jira/browse/ARROW-17535) / @emkornfield
#### Related issues:
- [[Python] Use ExtensionScalar.as_py() as fallback in ExtensionArray to_pandas?](https://github.com/apache/arrow/issues/33134) (relates to)
- [[Python] Nested ExtensionArray conversion to/from pandas/numpy](https://github.com/apache/arrow/issues/33036) (is related to)
**Note**: *This issue was originally created as [ARROW-17535](https://issues.apache.org/jira/browse/ARROW-17535). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by tracing the Python to_pandas path and the C++ handling of allowed types, then review ExtensionArray.to_pylist and the related ExtensionScalar.as_py fallback issue. Done means List arrays convert through to_pandas while preserving user-provided overrides; resolve the approach alongside the related nested ExtensionArray conversion issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100