`dataframe_to_arrow` Returns a table that doesn't convert geopandas index correctly
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 784
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 58
Description
A lot of text below, but I'll highlight the main difference first. Notice our version has extra nested `[ ]`.
```
# Our dataframe_to_arrow returns the following column
geometry: [[0101...F03F],[0101...0040]]
# But geopandas returns this.
geometry: [[0101...F03F,0101...0040]]
```
This happens for the index column (`__index_level_0__`) too, which leads to it being misterpreted as a column instead of being read in as an index when calling `gpd.GeoDataFrame.from_arrow()`
```
# Sedona returns
__index_level_0__ geometry
0 1 POINT (1 1)
1 2 POINT (2 2)
# Geopandas returns this
geometry
1 POINT (1 1)
2 POINT (2 2)
```
Full script and output below.
```python
import geopandas as gpd
import sedona.geopandas as sgpd
from sedona.spark.geoarrow.geoarrow import dataframe_to_arrow
sgpd_df = sgpd.GeoDataFrame({"geometry": [Point(1, 1), Point(2, 2)]}, index=pd.Index([1, 2]))
spark_df = sgpd_df._internal.spark_frame.drop("__natural_order__") # don't worry about this drop
sgpd_arrow = dataframe_to_arrow(spark_df)
gpd_df = gpd.GeoDataFrame({"geometry": [Point(1, 1), Point(2, 2)]}, index=pd.Index([1, 2]))
gpd_arrow = pa.table(gpd_df.to_arrow())
assert type(sgpd_arrow) == type(gpd_arrow) == pa.Table
print("SEDONA\n", sgpd_arrow, "\n")
gpd_df_from_sgpd_arrow = gpd.GeoDataFrame.from_arrow(sgpd_arrow)
print(gpd_df_from_sgpd_arrow, "\n")
print("GEOPANDAS\n", gpd_arrow, "\n")
gpd_df_from_gpd_arrow = gpd.GeoDataFrame.from_arrow(gpd_arrow)
print(gpd_df_from_gpd_arrow)
```
```
SEDONA
pyarrow.Table
__index_level_0__: int64
geometry: extension>
----
__index_level_0__: [[1],[2]]
geometry: [[0101000000000000000000F03F000000000000F03F],[010100000000000000000000400000000000000040]]
__index_level_0__ geometry
0 1 POINT (1 1)
1 2 POINT (2 2)
GEOPANDAS
pyarrow.Table
geometry: extension>
__index_level_0__: int64
----
geometry: [[0101000000000000000000F03F000000000000F03F,010100000000000000000000400000000000000040]]
__index_level_0__: [[1,2]]
geometry
1 POINT (1 1)
2 POINT (2 2)
```
Contributor guide
Research direction
Start at sedona.spark.geoarrow.geoarrow.dataframe_to_arrow and reproduce the issue with the provided GeoPandas and Spark DataFrame script. Compare the returned pyarrow.Table with GeoPandas' table, focusing on the nested geometry and __index_level_0__ arrays. Done means GeoDataFrame.from_arrow restores the index and geometry rows as shown by the GeoPandas result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100