apache / apache/sedona

`dataframe_to_arrow` Returns a table that doesn't convert geopandas index correctly

Open
#2,138 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.4k
Forks
784
Avg merge
1d 12h
Merged PRs (30d)
58

Description

A lot of text below, but I'll highlight the main difference first. Notice our version has extra nested `[ ]`.
```
# Our dataframe_to_arrow returns the following column
geometry: [[0101...F03F],[0101...0040]]

# But geopandas returns this.
geometry: [[0101...F03F,0101...0040]]
```

This happens for the index column (`__index_level_0__`) too, which leads to it being misterpreted as a column instead of being read in as an index when calling `gpd.GeoDataFrame.from_arrow()`
```
# Sedona returns
__index_level_0__ geometry
0 1 POINT (1 1)
1 2 POINT (2 2)

# Geopandas returns this
geometry
1 POINT (1 1)
2 POINT (2 2)
```

Full script and output below.
```python
import geopandas as gpd
import sedona.geopandas as sgpd
from sedona.spark.geoarrow.geoarrow import dataframe_to_arrow

sgpd_df = sgpd.GeoDataFrame({"geometry": [Point(1, 1), Point(2, 2)]}, index=pd.Index([1, 2]))
spark_df = sgpd_df._internal.spark_frame.drop("__natural_order__") # don't worry about this drop
sgpd_arrow = dataframe_to_arrow(spark_df)

gpd_df = gpd.GeoDataFrame({"geometry": [Point(1, 1), Point(2, 2)]}, index=pd.Index([1, 2]))
gpd_arrow = pa.table(gpd_df.to_arrow())
assert type(sgpd_arrow) == type(gpd_arrow) == pa.Table
print("SEDONA\n", sgpd_arrow, "\n")
gpd_df_from_sgpd_arrow = gpd.GeoDataFrame.from_arrow(sgpd_arrow)
print(gpd_df_from_sgpd_arrow, "\n")
print("GEOPANDAS\n", gpd_arrow, "\n")
gpd_df_from_gpd_arrow = gpd.GeoDataFrame.from_arrow(gpd_arrow)
print(gpd_df_from_gpd_arrow)
```

```
SEDONA
pyarrow.Table
__index_level_0__: int64
geometry: extension>
----
__index_level_0__: [[1],[2]]
geometry: [[0101000000000000000000F03F000000000000F03F],[010100000000000000000000400000000000000040]]

__index_level_0__ geometry
0 1 POINT (1 1)
1 2 POINT (2 2)

GEOPANDAS
pyarrow.Table
geometry: extension>
__index_level_0__: int64
----
geometry: [[0101000000000000000000F03F000000000000F03F,010100000000000000000000400000000000000040]]
__index_level_0__: [[1,2]]

geometry
1 POINT (1 1)
2 POINT (2 2)
```

Contributor guide

Open the contributing guide

Research direction

Start at sedona.spark.geoarrow.geoarrow.dataframe_to_arrow and reproduce the issue with the provided GeoPandas and Spark DataFrame script. Compare the returned pyarrow.Table with GeoPandas' table, focusing on the nested geometry and __index_level_0__ arrays. Done means GeoDataFrame.from_arrow restores the index and geometry rows as shown by the GeoPandas result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.