apache / apache/sedona-db

Raster execution with materialized values needs adaptive batch sizing to prevent out of memory

Open
#1,220 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 5h
Merged PRs (30d)
91

Description

Some initial attempts at implementing a spatialbench Raster run have resulted in immediate OOMs for sf1, which is expected because we use the default batch size (8192) and materializing 8000 rasters is almost always a bad idea.

When we have an operation that will allocate a new raster for every row (e.g., ensure loaded or rasterize), we need to adaptively shrink the batch size based on some heuristic of number of bytes that will be allocated. I think this will be a physical optimizer rule inserted before a projection or filter, and in its initial form it could be crude (just make the batches defensively smaller when a projection contains a raster operation).

Contributor guide

Open the contributing guide

Research direction

Start with the spatialbench Raster run and the default batch size of 8192, then trace how projections and filters are planned and where a physical optimizer rule could be inserted. Done means raster operations such as ensure loaded or rasterize use smaller batches based on allocation needs and no longer immediately run out of memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.