Unexpected performance querying geoparquet vs parquet
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 784
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 58
Description
Hi
I'm trying to setup Sedona to run spatial intersection queries on a multi-file geoparquet 1.1.0 dataset that I've generated with GeoPandas. The total dataset size is approximately 2.5Gb split across 4 files.
`dataframe.to_parquet(outfile, write_covering_bbox=True, schema_version='1.1.0')`
I'm seeing some unexpected behaviour where reading the file in as a parquet file results in much better performance relative to geoparquet. When I load the dataframe in as parquet and then create the geometry, my query completes in ~3-5s. When reading in natively as geoparquet it takes ~15s. In both cases this is the query I've run.
`SELECT * FROM water_with_geom_(g)pq where ST_Intersects(ST_GeomFromWKT('POLYGON ((411908 128831, 411927 133556, 416895 134004, 417044 128326, 411908 128831))', 27700), geometry)`
My GeoParquet workflow is:
```
val geo_df = sedona.read.format("geoparquet").load("dbfs:/FileStore/tables/geoparquet_investigation/water")
geo_df.createOrReplaceTempView("water_with_geom_gpq")
```
And for Parquet:
```
val df = sedona.read.format("parquet").load("dbfs:/FileStore/tables/geoparquet_investigation/water")
df.createOrReplaceTempView("temp_water")
val geom_df = sedona.sql("SELECT *, ST_GeomFromWKB(geometry) as geom from temp_water")
val columnsToDrop = Seq("geometry")
val geom_df_dropped = geom_df.drop(columnsToDrop: _*)
val geometry_df = geom_df_dropped.withColumnRenamed("geom", "geometry")
geometry_df.createOrReplaceTempView("water_with_geom_pq")
```
My environment is a databricks cluster (DBR 15.4, Spark 3.5, Scala 2.12) running Sedona 1.7.0
Would greatly appreciate if someone could point out where I'm going wrong - am very new to Sedona and fairly new to all things Spark!
Contributor guide
Research direction
Reproduce the two workflows on the referenced four-file dataset in the stated Databricks, Spark 3.5, and Sedona 1.7.0 environment, comparing native geoparquet loading with parquet loading followed by ST_GeomFromWKB. Inspect the execution plans and timings for the ST_Intersects query; done means identifying the cause of the performance difference or documenting the missing optimization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scala, spark
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100