apache / apache/sedona

Save / Load indexed spatial & partitioned Rdd

Open
#1,213 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.4k
Forks
784
Avg merge
1d 12h
Merged PRs (30d)
58

Description

## Expected behavior

Maybe this is possible somehow, but I haven't find this anywhere. I'm relatively new to Sedona and Geo-processing.
I'd like to see a possibility to save and then load a spatial RDD which is already analyzed, partitioned and possibly with the index. We have a use case we use such dataset in many jobs (which use the same spatial data) and it's time-consuming to create the partitioning & build index every time.
Not sure if it's possible though.

For example:

```
// save once:
val spatialRdd = Adapter.toSpatialRdd(df, ...)
spatialRdd.analyze()
spatialRdd.spatialPartitioning(GridType.KDBTREE, math.min(Integer.MAX_VALUE, df.count() / 2).toInt) // IllegalArgumentException: [Sedona] Number of partitions cannot be larger than half of total records num
spatialRdd.buildIndex(IndexType.RTREE, true)
SomeSedonaUtility.saveSpatialRdd(spatialRdd, path) // <-- save with index and partitioned

// load & use multiple times:
val rdd = SomeSedonaUtility.loadSpatialRdd(path)

// and usage:
val otherRdd = Adapter.toSpatialRdd(otherDs, ...)
otherRdd.spatialPartitioning(rdd.getPartitioner)

val useIndex = true
val considerBoundaryIntersection = SpatialPredicate.COVERS
val params = new JoinQuery.JoinParams(useIndex, considerBoundaryIntersection, IndexType.RTREE, JoinBuildSide.LEFT)

val joined = JoinQuery.spatialJoin(rdd, otherRdd, params)
```

## Actual behavior

Index & partitioning must be set at runtime (to my knowledge).

## Steps to reproduce the problem

The feature is missing, so it's not possible to reproduce it.

## Settings

Sedona version = 1.5.1

Apache Spark version = 3.5

API type = Scala

Scala version = 2.12

JRE version = 1.8

Environment = EMR

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing Adapter.toSpatialRdd, spatialPartitioning, buildIndex, and JoinQuery APIs to understand which analyzed, partitioned, and indexed state can be retained. Define how a saved spatial RDD would be loaded and reused across jobs, including its partitioner and index, then verify that the loaded RDD works with the shown spatial join flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.