[Python] Cannot convert pd.DataFrame with geometry cells to pa.Table
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Example:
```java
import geopandas as gpd
import pandas as pd
import pyarrow as pa
path = gpd.datasets.get_path("naturalearth_lowres")
data = gpd.read_file(path)
df = pd.DataFrame(data)
table = pa.Table.from_pandas(df)
print(table)
```
Throws the following error:
```java
Traceback (most recent call last):
File "/Users/Henrikh/Desktop/tmp.py", line 8, in
table = pa.Table.from_pandas(df)
File "pyarrow/table.pxi", line 1553, in pyarrow.lib.Table.from_pandas
File "/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 594, in dataframe_to_arrays
arrays = [convert_column(c, f)
File "/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 594, in
arrays = [convert_column(c, f)
File "/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 581, in convert_column
raise e
File "/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 575, in convert_column
result = pa.array(col, type=type_, from_pandas=True, safe=safe)
File "pyarrow/array.pxi", line 302, in pyarrow.lib.array
File "pyarrow/array.pxi", line 79, in pyarrow.lib._ndarray_to_array
File "pyarrow/array.pxi", line 67, in pyarrow.lib._ndarray_to_type
File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column geometry with type geometry')
```
**Reporter**: [Henrikh Kantuni](https://issues.apache.org/jira/browse/ARROW-14267)
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-14267) / @rok
**Note**: *This issue was originally created as [ARROW-14267](https://issues.apache.org/jira/browse/ARROW-14267). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with pa.Table.from_pandas and the pandas_compat.py conversion path shown in the traceback, then reproduce the example using the GeoDataFrame-derived pandas DataFrame. Investigate how the geometry column reaches pa.array; done means the example converts successfully without the ArrowTypeError and the behavior is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100