Save / Load indexed spatial & partitioned Rdd
- Dominant language
- Java
- Stars
- 2.4k
- Forks
- 784
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 58
Description
## Expected behavior
Maybe this is possible somehow, but I haven't find this anywhere. I'm relatively new to Sedona and Geo-processing.
I'd like to see a possibility to save and then load a spatial RDD which is already analyzed, partitioned and possibly with the index. We have a use case we use such dataset in many jobs (which use the same spatial data) and it's time-consuming to create the partitioning & build index every time.
Not sure if it's possible though.
For example:
```
// save once:
val spatialRdd = Adapter.toSpatialRdd(df, ...)
spatialRdd.analyze()
spatialRdd.spatialPartitioning(GridType.KDBTREE, math.min(Integer.MAX_VALUE, df.count() / 2).toInt) // IllegalArgumentException: [Sedona] Number of partitions cannot be larger than half of total records num
spatialRdd.buildIndex(IndexType.RTREE, true)
SomeSedonaUtility.saveSpatialRdd(spatialRdd, path) // <-- save with index and partitioned
// load & use multiple times:
val rdd = SomeSedonaUtility.loadSpatialRdd(path)
// and usage:
val otherRdd = Adapter.toSpatialRdd(otherDs, ...)
otherRdd.spatialPartitioning(rdd.getPartitioner)
val useIndex = true
val considerBoundaryIntersection = SpatialPredicate.COVERS
val params = new JoinQuery.JoinParams(useIndex, considerBoundaryIntersection, IndexType.RTREE, JoinBuildSide.LEFT)
val joined = JoinQuery.spatialJoin(rdd, otherRdd, params)
```
## Actual behavior
Index & partitioning must be set at runtime (to my knowledge).
## Steps to reproduce the problem
The feature is missing, so it's not possible to reproduce it.
## Settings
Sedona version = 1.5.1
Apache Spark version = 3.5
API type = Scala
Scala version = 2.12
JRE version = 1.8
Environment = EMR
Contributor guide
Research direction
Start by reviewing the existing Adapter.toSpatialRdd, spatialPartitioning, buildIndex, and JoinQuery APIs to understand which analyzed, partitioned, and indexed state can be retained. Define how a saved spatial RDD would be loaded and reused across jobs, including its partitioner and index, then verify that the loaded RDD works with the shown spatial join flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100