Raster execution with materialized values needs adaptive batch sizing to prevent out of memory
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 91
Description
Some initial attempts at implementing a spatialbench Raster run have resulted in immediate OOMs for sf1, which is expected because we use the default batch size (8192) and materializing 8000 rasters is almost always a bad idea.
When we have an operation that will allocate a new raster for every row (e.g., ensure loaded or rasterize), we need to adaptively shrink the batch size based on some heuristic of number of bytes that will be allocated. I think this will be a physical optimizer rule inserted before a projection or filter, and in its initial form it could be crude (just make the batches defensively smaller when a projection contains a raster operation).
Contributor guide
Research direction
Start with the spatialbench Raster run and the default batch size of 8192, then trace how projections and filters are planned and where a physical optimizer rule could be inserted. Done means raster operations such as ensure loaded or rasterize use smaller batches based on allocation needs and no longer immediately run out of memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100