apache / apache/arrow

[Python] Use ExtensionScalar.as_py() as fallback in ExtensionArray to_pandas?

Open
#33,134 6 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

This was raised in ARROW-17813 by @changhiskhan:

> **ExtensionArray => pandas**
>
> Just for discussion, I was curious whether you had any thoughts around using the extension scalar as a fallback mechanism. It's a lot simpler to define an ExtensionScalar with `as_py` than a pandas extension dtype. So if an ExtensionArray doesn't have an equivalent pandas dtype, would it make sense to convert it to just an object series whose elements are the result of `as_py`?

and I also mentioned this in ARROW-17535:

> That actually brings up a question: if an ExtensionType defines an ExtensionScalar (but not an associciated pandas dtype, or custom to_numpy conversion), should we use this scalar's `as_py()` for the to_numpy/to_pandas conversion as well for plain extension arrays? (not the nested case)
>
> Because currently, if you have an ExtensionArray like that (for example using the example from the docs: https://arrow.apache.org/docs/dev/python/extending_types.html#custom-scalar-conversion), we still use the storage type conversion for to_numpy/to_pandas, and only use the scalar's conversion in `to_pylist`.

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-17925) / @jorisvandenbossche
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-17925) / @rok
#### Related issues:
- [[Python] List arrays aren't supported in to_pandas calls](https://github.com/apache/arrow/issues/32791) (is related to)
- [[Python] Nested ExtensionArray conversion to/from pandas/numpy](https://github.com/apache/arrow/issues/33036) (is related to)

**Note**: *This issue was originally created as [ARROW-17925](https://issues.apache.org/jira/browse/ARROW-17925). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the custom scalar conversion documentation linked in the issue, then read the related issues ARROW-17813, ARROW-17535, #32791, and #33036 for the intended conversion boundaries. Determine the expected behavior for plain ExtensionArray to_numpy and to_pandas conversions, including the object-series case; done means the behavior is covered for the stated non-nested scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.