apache / apache/sedona-db

Add cache layer for raster loading

Open
#1,224 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 4h
Merged PRs (30d)
90

Description

The current raster loader (or any of the implementations) make no attempt to cache multiple requests to the same object, or any version of a cache (except possibly a dataset cache in the GDAL layer, whose details I forget). In Sedona Spark I believe there are at least two levels of caching, although the per-row execution may be better suited to effectively using it (as opposed to the current execution that loops through potentially 8000 rasters before circling back to a new expression evaluation). We should implement some version of a cache before we get too far into evaluation to make sure the APIs between the loader and their implementations effectively capture an effective cache scheme.

Contributor guide

Open the contributing guide

Research direction

Start by locating the current raster loader and its implementations, then review how repeated object requests are evaluated. Compare the stated Sedona Spark caching levels and the possible GDAL dataset cache before defining the loader/implementation cache API. Done means an agreed cache scheme is implemented and repeated raster loads avoid unnecessary work.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.