uber / uber/h3-java

H3Core.nonZeroLongArrayToList takes too much memory

Open
#68 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
333
Forks
68
PR merge metrics
No merged PRs in 30d

Description

Trying to polyfill big area with resolution 11. I can see that internally algorithm prepared result https://github.com/uber/h3-java/blob/3a1e9bc9bcd1d62e9e8d4361230fe47bb5395242/src/main/java/com/uber/h3core/H3Core.java#L691 but when it tries to copy it to List it gets out of memory on this line:
https://github.com/uber/h3-java/blob/a500880eef493dca44903bf7ad0e249aed5b85b1/src/main/java/com/uber/h3core/H3Core.java#L1242-L1254

As you can see in the stacktrace, result is ready, but it just tries to copy it to ArrayList:
image

Memory, the arrow shows to the place when nonZeroLongArrayToList started :
image

Scala code:

import scala.collection.JavaConverters._
import java.util.{Collections => JCollections}

import beam.utils.ProfilingUtils
import com.uber.h3core.AreaUnit
import com.uber.h3core.util.GeoCoord

def main(args: Array[String]): Unit = {

  val h3Core = com.uber.h3core.H3Core.newInstance

  val xMin = -106.645646
  val xMax = -93.508292
  val yMin = 25.837377
  val yMax = 36.500704

  val rectangle = java.util.Arrays.asList(
    new GeoCoord(yMin, xMin),
    new GeoCoord(yMax, xMin),
    new GeoCoord(yMax, xMax),
    new GeoCoord(yMin, xMax),
  )

  val holes = java.util.Collections.emptyList[java.util.List[GeoCoord]]()
  val resolution = 11
  val hexes = h3Core.polyfillAddress(rectangle, holes, resolution)
  println(s"Generated ${hexes.size()}")
}

The easiest fix is just to get the total number of non-zero elements and allocate an array with that size and copy elements over. Other solution can be moving all zero elements in the original array to the end of the array and wrap it by ArrayList via Arrays.asList(array).subList(index, IDX_OF_FIRST_ZERO);. I can create PR if this sounds good.

Thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/main/java/com/uber/h3core/H3Core.java at the polyfill result preparation around line 691 and nonZeroLongArrayToList around lines 1242-1254. Rework the conversion so the prepared non-zero result does not require an oversized intermediate ArrayList, then verify that polyfillAddress preserves the generated hexes without the reported out-of-memory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.