JuliaDataCubes / JuliaDataCubes/EarthDataLab.jl
cubefromshape fails with samplefactor due to OutOfMemory Error
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 34
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
When I try to use cubefromshape with a samplefactor, this fails with an OutOfMemory Error, because the cube is too large to have the fraction value for every pixel in memory.
One Possible solution:
We could circumvent this by building a ShapeCube type which saves the boundaries of the polygons or maybe the intersections of the polygons with latitude lines at the pixel centers of the cube or of a resampled cube and then we can put the computation of whether a pixel is inside the polygon or outside of it into the getindex function of the ShapeCube type.
This would make the getindex function calls of such a cube a little bit slower, but it would make the memory consumption of this much lower and this would make it possible to save larger polygons of larger cubes in a ShapeCube.
And if the computation is a problem, than one could use the savecube function to save the ShapeCube on disk, because this saving should compute the value for every pixel, because we are going to save the full cube on disk. And then one could have the actual values and the getindex calls would be as fast as a normal cube.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the cubefromshape path and reproduce the reported samplefactor case that raises OutOfMemoryError. Read the proposed ShapeCube, getindex, and savecube design in the issue, then determine how polygon boundaries or intersections can be retained without storing a fraction value for every pixel. Done means large shape cubes no longer exhaust memory while indexing and saving still produce the expected values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100