Add cache layer for raster loading
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
The current raster loader (or any of the implementations) make no attempt to cache multiple requests to the same object, or any version of a cache (except possibly a dataset cache in the GDAL layer, whose details I forget). In Sedona Spark I believe there are at least two levels of caching, although the per-row execution may be better suited to effectively using it (as opposed to the current execution that loops through potentially 8000 rasters before circling back to a new expression evaluation). We should implement some version of a cache before we get too far into evaluation to make sure the APIs between the loader and their implementations effectively capture an effective cache scheme.
Contributor guide
Research direction
Start by locating the current raster loader and its implementations, then review how repeated object requests are evaluated. Compare the stated Sedona Spark caching levels and the possible GDAL dataset cache before defining the loader/implementation cache API. Done means an agreed cache scheme is implemented and repeated raster loads avoid unnecessary work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100