locationtech / locationtech/geotrellis

SpacePartitioner distributes everything to just a few partitions/executors

Open
#3,274 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug question
Dominant language
Scala
Stars
1.4k
Forks
360
Avg merge
6h 57m
Merged PRs (30d)
12

Description

Describe the bug

I might just be using this wrong, because I can't find much documentation at all on how to use the SpacePartitioner. I'm trying to use the SpacePartitioner on an RDD of type RDD[(SpatialKey, geotrellis.vector.Feature)]. I've created a LayoutDefinition that covers the entire pan-tropics (so 180E-180W and 30S-30N), and vector features cover are land in those bounds, so I just set the KeyBounds to the entire layout.

But even though I'm running with ~700 executors, it's giving everything to just 3 executors (vs. when I use HashPartitioner, where it distributes to pretty much all executors). I'm not sure if I'm just using it incorrectly (my bounds seem wrong) or it just doesn't support global-scale partitioning that well (this post makes it seem like it's intended for more local areas with clear bounding boxes).

Environment

  • Scala version: 2.11.12
  • Spark version: 2.4.0
  • GeoTrellis version: 2.2.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing how SpacePartitioner uses the LayoutDefinition and full-layout KeyBounds for the RDD[(SpatialKey, geotrellis.vector.Feature)] case. Compare its distribution with HashPartitioner at the reported global bounds and consult issue #1206; done means determining whether the behavior is incorrect or documenting the supported usage and expected partition distribution.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.