GenericMappingTools / GenericMappingTools/pygmt
Support PyArrow arrays and dataframes
- Dominant language
- Python
- Stars
- 874
- Forks
- 255
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 40
Description
### Description of the desired feature
[Apache Arrow](https://arrow.apache.org/docs/index.html) is an in-memory format that is starting to become a common exchange format between different libraries in Python and other programming languages. For example:
- Pandas 3.0 will use `pyarrow.string` instead of `object` dtype for strings, see [PDEP10](https://pandas.pydata.org/pdeps/0010-required-pyarrow-dependency.html), so we will eventually need to support PyArrow (at least for string dtypes)
- [Polars](https://github.com/pola-rs/polars) DataFrames can be zero-copy converted to Pyarrow via the `__dataframe__` protocol, see https://arrow.apache.org/docs/python/interchange_protocol.html
This issue is to track compatibility and support of different [PyArrow data types](https://arrow.apache.org/docs/python/api/datatypes.html) in PyGMT:
| Dtype | Implementation PR | Status | Notes |
|--|--|--|--|
| Numerical (uint/int/float) | #2774 | :white_check_mark: | |
| String | #2933 | :construction: | May require modifying the [`put_strings`](https://github.com/GenericMappingTools/pygmt/blob/v0.10.0/pygmt/clib/session.py#L917) method that currently uses `np.char.encode` |
| Date/Time | #2845 (pandas) and TODO (raw pyarrow) | :construction: | May require modifying [`array_to_datetime`](https://github.com/GenericMappingTools/pygmt/blob/v0.10.0/pygmt/clib/conversion.py#L252) that expects Python datetime or numpy-backed arrays, xref #242 and #3507 |
| Duration | TODO | :x: | https://arrow.apache.org/docs/13.0/python/generated/pyarrow.duration.html, wait for #2884 also |
| Special case: `geopandas.GeoDataFrame` with PyArrow dtype columns | TODO | :x: | See https://github.com/GenericMappingTools/pygmt/pull/2774#discussion_r1413006621 |
| GeoArrow geometry | TODO | :x: | https://github.com/geoarrow/geoarrow-python |
Simplest way of integrating would be to just handle PyArrow-backed `pandas.Dataframe` objects as above.
Alternatively, we can also discuss about using PyArrow as the internal array representation (which would make `pyarrow` a hard dependency) since it may allow better interoperability across other Python libraries using Arrow, and this might be relevant for #1318 and #2731. My thought is to do this through the [`__dataframe__` protocol](https://data-apis.org/dataframe-protocol/latest/purpose_and_scope.html), see https://arrow.apache.org/docs/python/interchange_protocol.html
Further reading:
- https://voltrondata.com/codex/standards-over-silos#1-2-standardizing-on-arrow
- https://voltrondata.com/resources/dataframe-interoperability-python-pyarrow-enables-modular-workflows
- https://arrow.apache.org/docs/python/pandas.html#zero-copy-series-conversions
### Are you willing to help implement and maintain this feature?
Yes, but help is welcome too!
Contributor guide
Assessment
This issue has not been assessed yet.