apache / apache/arrow

[Python] Cannot convert pd.DataFrame with geometry cells to pa.Table

Open
#29,844 3 comments 6 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Example: 
```java

import geopandas as gpd
import pandas as pd
import pyarrow as pa

path = gpd.datasets.get_path("naturalearth_lowres")
data = gpd.read_file(path)
df = pd.DataFrame(data)
table = pa.Table.from_pandas(df)
print(table)
```
Throws the following error:
```java

Traceback (most recent call last):
File "/Users/Henrikh/Desktop/tmp.py", line 8, in
table = pa.Table.from_pandas(df)
File "pyarrow/table.pxi", line 1553, in pyarrow.lib.Table.from_pandas
File "/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 594, in dataframe_to_arrays
arrays = [convert_column(c, f)
File "/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 594, in
arrays = [convert_column(c, f)
File "/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 581, in convert_column
raise e
File "/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 575, in convert_column
result = pa.array(col, type=type_, from_pandas=True, safe=safe)
File "pyarrow/array.pxi", line 302, in pyarrow.lib.array
File "pyarrow/array.pxi", line 79, in pyarrow.lib._ndarray_to_array
File "pyarrow/array.pxi", line 67, in pyarrow.lib._ndarray_to_type
File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column geometry with type geometry')
```
 

**Reporter**: [Henrikh Kantuni](https://issues.apache.org/jira/browse/ARROW-14267)
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-14267) / @rok

**Note**: *This issue was originally created as [ARROW-14267](https://issues.apache.org/jira/browse/ARROW-14267). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with pa.Table.from_pandas and the pandas_compat.py conversion path shown in the traceback, then reproduce the example using the GeoDataFrame-derived pandas DataFrame. Investigate how the geometry column reaches pa.array; done means the example converts successfully without the ArrowTypeError and the behavior is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.