apache / apache/arrow

[Python] Convert maps to dicts without per-element Scalars in to_pylist(maps_as_pydicts=...)

Open
#50,429 0 comments 0 reactions 1 assignee Claimed by @viirya View on GitHub
Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Follow-up of #50326. GH-50327 makes `Array.to_pylist()` convert without per-element Scalars, but the `maps_as_pydicts` option still routes to the Scalar-based path: every map row allocates a `MapScalar`, converts its keys via per-element `as_py`, and builds the dict in Python.

Converting maps to dicts is the natural consumption pattern for engines whose map values are Python dicts (e.g. Spark's Arrow-serialized Python UDFs currently receive association lists from `to_pylist()` and rebuild a dict per row in pure Python — one of the dominant remaining costs in that path).

Proposal: thread `maps_as_pydicts` through the scalar-free `_getitem_py` mechanism introduced in GH-50327 so `MapArray` builds the dict directly from the flattened keys/items children:

- default (`None`) semantics unchanged (association lists);
- `'lossy'`/`'strict'` build the dict in one pass; when the dict size shows duplicate keys, the row is redone with the careful per-key loop so the warning ('lossy') and `KeyError` ('strict') semantics stay exactly identical to `MapScalar.as_py`;
- invalid values still raise the same `ValueError` when a map value is converted;
- the option propagates through nested types (list/struct/map values) as before, and unspecialized types keep the exact Scalar fallback.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.