JuliaDataCubes / JuliaDataCubes/EarthDataLab.jl

cubefromshape fails with samplefactor due to OutOfMemory Error

Open
#203 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
34
Forks
14
PR merge metrics
No merged PRs in 30d

Description

When I try to use cubefromshape with a samplefactor, this fails with an OutOfMemory Error, because the cube is too large to have the fraction value for every pixel in memory.

One Possible solution:
We could circumvent this by building a ShapeCube type which saves the boundaries of the polygons or maybe the intersections of the polygons with latitude lines at the pixel centers of the cube or of a resampled cube and then we can put the computation of whether a pixel is inside the polygon or outside of it into the getindex function of the ShapeCube type.
This would make the getindex function calls of such a cube a little bit slower, but it would make the memory consumption of this much lower and this would make it possible to save larger polygons of larger cubes in a ShapeCube.
And if the computation is a problem, than one could use the savecube function to save the ShapeCube on disk, because this saving should compute the value for every pixel, because we are going to save the full cube on disk. And then one could have the actual values and the getindex calls would be as fast as a normal cube.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the cubefromshape path and reproduce the reported samplefactor case that raises OutOfMemoryError. Read the proposed ShapeCube, getindex, and savecube design in the issue, then determine how polygon boundaries or intersections can be retained without storing a fraction value for every pixel. Done means large shape cubes no longer exhaust memory while indexing and saving still produce the expected values.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.