GenericMappingTools / GenericMappingTools/pygmt

Support PyArrow arrays and dataframes

Open
#2,800 3 comments 1 reaction 1 assignee Claimed by @weiji14 View on GitHub
feature request help wanted longterm
Dominant language
Python
Stars
874
Forks
255
Avg merge
1d 21h
Merged PRs (30d)
40

Description

### Description of the desired feature

[Apache Arrow](https://arrow.apache.org/docs/index.html) is an in-memory format that is starting to become a common exchange format between different libraries in Python and other programming languages. For example:

- Pandas 3.0 will use `pyarrow.string` instead of `object` dtype for strings, see [PDEP10](https://pandas.pydata.org/pdeps/0010-required-pyarrow-dependency.html), so we will eventually need to support PyArrow (at least for string dtypes)
- [Polars](https://github.com/pola-rs/polars) DataFrames can be zero-copy converted to Pyarrow via the `__dataframe__` protocol, see https://arrow.apache.org/docs/python/interchange_protocol.html

This issue is to track compatibility and support of different [PyArrow data types](https://arrow.apache.org/docs/python/api/datatypes.html) in PyGMT:

| Dtype | Implementation PR | Status | Notes |
|--|--|--|--|
| Numerical (uint/int/float) | #2774 | :white_check_mark: | |
| String | #2933 | :construction: | May require modifying the [`put_strings`](https://github.com/GenericMappingTools/pygmt/blob/v0.10.0/pygmt/clib/session.py#L917) method that currently uses `np.char.encode` |
| Date/Time | #2845 (pandas) and TODO (raw pyarrow) | :construction: | May require modifying [`array_to_datetime`](https://github.com/GenericMappingTools/pygmt/blob/v0.10.0/pygmt/clib/conversion.py#L252) that expects Python datetime or numpy-backed arrays, xref #242 and #3507 |
| Duration | TODO | :x: | https://arrow.apache.org/docs/13.0/python/generated/pyarrow.duration.html, wait for #2884 also |
| Special case: `geopandas.GeoDataFrame` with PyArrow dtype columns | TODO | :x: | See https://github.com/GenericMappingTools/pygmt/pull/2774#discussion_r1413006621 |
| GeoArrow geometry | TODO | :x: | https://github.com/geoarrow/geoarrow-python |

Simplest way of integrating would be to just handle PyArrow-backed `pandas.Dataframe` objects as above.

Alternatively, we can also discuss about using PyArrow as the internal array representation (which would make `pyarrow` a hard dependency) since it may allow better interoperability across other Python libraries using Arrow, and this might be relevant for #1318 and #2731. My thought is to do this through the [`__dataframe__` protocol](https://data-apis.org/dataframe-protocol/latest/purpose_and_scope.html), see https://arrow.apache.org/docs/python/interchange_protocol.html

Further reading:
- https://voltrondata.com/codex/standards-over-silos#1-2-standardizing-on-arrow
- https://voltrondata.com/resources/dataframe-interoperability-python-pyarrow-enables-modular-workflows
- https://arrow.apache.org/docs/python/pandas.html#zero-copy-series-conversions

### Are you willing to help implement and maintain this feature?

Yes, but help is welcome too!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.