apache / apache/sedona

Unexpected performance querying geoparquet vs parquet

Open
#1,845 6 comments 0 reactions 0 assignees View on GitHub
affect public APIs
Dominant language
Java
Stars
2.4k
Forks
784
Avg merge
1d 12h
Merged PRs (30d)
58

Description

Hi

I'm trying to setup Sedona to run spatial intersection queries on a multi-file geoparquet 1.1.0 dataset that I've generated with GeoPandas. The total dataset size is approximately 2.5Gb split across 4 files.

`dataframe.to_parquet(outfile, write_covering_bbox=True, schema_version='1.1.0')`

I'm seeing some unexpected behaviour where reading the file in as a parquet file results in much better performance relative to geoparquet. When I load the dataframe in as parquet and then create the geometry, my query completes in ~3-5s. When reading in natively as geoparquet it takes ~15s. In both cases this is the query I've run.

`SELECT * FROM water_with_geom_(g)pq where ST_Intersects(ST_GeomFromWKT('POLYGON ((411908 128831, 411927 133556, 416895 134004, 417044 128326, 411908 128831))', 27700), geometry)`

My GeoParquet workflow is:

```
val geo_df = sedona.read.format("geoparquet").load("dbfs:/FileStore/tables/geoparquet_investigation/water")

geo_df.createOrReplaceTempView("water_with_geom_gpq")
```

And for Parquet:

```
val df = sedona.read.format("parquet").load("dbfs:/FileStore/tables/geoparquet_investigation/water")
df.createOrReplaceTempView("temp_water")
val geom_df = sedona.sql("SELECT *, ST_GeomFromWKB(geometry) as geom from temp_water")
val columnsToDrop = Seq("geometry")
val geom_df_dropped = geom_df.drop(columnsToDrop: _*)
val geometry_df = geom_df_dropped.withColumnRenamed("geom", "geometry")
geometry_df.createOrReplaceTempView("water_with_geom_pq")
```

My environment is a databricks cluster (DBR 15.4, Spark 3.5, Scala 2.12) running Sedona 1.7.0

Would greatly appreciate if someone could point out where I'm going wrong - am very new to Sedona and fairly new to all things Spark!

Contributor guide

Open the contributing guide

Research direction

Reproduce the two workflows on the referenced four-file dataset in the stated Databricks, Spark 3.5, and Sedona 1.7.0 environment, comparing native geoparquet loading with parquet loading followed by ST_GeomFromWKB. Inspect the execution plans and timings for the ST_Intersects query; done means identifying the cause of the performance difference or documenting the missing optimization.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scala, spark
Domain
data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.