RS_EnsureLoaded / AsyncByteLoader: round-trip non-identity band views (view in -> realized view out)
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
## Background
`RS_EnsureLoaded` materialises OutDb band bytes and passes InDb bands through. Today it cannot handle a band with a **non-identity view** (any slice / crop / broadcast / permutation, where the visible `shape()` differs from the backing `raw_source_shape()`):
- `ensure_loaded` rebuilds each band field-by-field via `start_band_nd` and never copies `band.view()`, so the view is silently dropped.
- `OutDbLoadRequest` carries only `source_shape` (the full backing extent); a loader has no way to learn that a sub-window was wanted.
- `AsyncByteLoader::load(req) -> Buffer` returns bare bytes; the caller can't tell what region/layout they represent.
This is **latent, not live**: `RasterRefImpl::band()` currently rejects non-identity views (`"view composition is not yet implemented"`), so such bands are unconstructible. The guardrail must come off as part of view-composition work (#813) — and when it does, `RS_EnsureLoaded` will silently corrupt viewed rasters unless this is addressed. **This issue is a correctness co-requisite of view composition landing.**
## The contract change: view in → realized view out
If the request carries a desired view (so a cloud loader *can* range-read just the sub-window instead of fetching a whole chunk), the loader has a legitimate choice, and the caller can't infer which it made from a bare buffer:
| Loader behavior | buffer | realized source_shape | realized view |
|---|---|---|---|
| Ignores crop (atomic chunk/tile — today's backends) | full backing bytes | full backing shape | input's view (unchanged) |
| Honors crop, returns contiguous cropped bytes | smaller | cropped shape | identity over cropped shape |
| Honors crop, returns full layout + selecting view | full | full | crop view |
`source_shape` and `view` move independently across these rows and neither is derivable from the buffer (its length only constrains `Π source_shape × byte_size`; a view entry doesn't reveal its source axis's true length). So the loader must report **all three** — the response is the data-bearing portion of a band:
```rust
struct LoadedBand {
buffer: Buffer,
source_shape: Vec, // natural C-order extent of `buffer`
view: Vec, // how visible axes map onto source_shape
}
async fn load(&self, req: &OutDbLoadRequest<'_>) -> Result;
```
`OutDbLoadRequest` gains the desired `view` (advisory — a loader may ignore it and return the full backing). Every loader, even the atomic ones, returns `source_shape` + an echoed/identity view, not just bytes.
## Seam with the builder copy work (#896)
With `copy_band_from` (#896), `RS_EnsureLoaded` becomes copy-the-metadata, override the loader-controlled layout:
```rust
copy_band_from(input) // name, dim_names, dtype, nodata, outdb_uri/format
.with_data_layout(resp.buffer, resp.source_shape, resp.view);
```
The loader owns exactly the three things a partial read can change (`buffer`, `source_shape`, `view`); the builder copy owns everything else. "Ignored the crop" is just the special case where the realized view/shape equal the input's.
## Scope
- `OutDbLoadRequest`: add the desired `view`.
- `AsyncByteLoader::load`: return `LoadedBand { buffer, source_shape, view }` instead of `Buffer`; update GDAL + Zarr loaders to echo `source_shape` + identity view (they ignore the crop for now).
- `RS_EnsureLoaded`: carry the input band's view into the request; build output bands from the realized response (via #896's copy + layout override).
- Remove the `RasterRefImpl::band()` non-identity-view rejection as the view-composition machinery (#813) makes them constructible (coordinate with that workstream).
## Not in scope
- Actually *implementing* crop pushdown in any loader (cloud range-reads). This issue only ensures the request/response can express it and that `RS_EnsureLoaded` stays correct when views exist.
## Related
- #813 (view composition / `RasterBuilder::with_view` — the forcing function; co-requisite)
- #896 (RasterBuilder copy APIs — the mechanism for the round-trip)
- #894 (zero-copy band-data path)
Contributor guide
Research direction
Start with OutDbLoadRequest, AsyncByteLoader::load, and RS_EnsureLoaded, then review the related view-composition work in #813 and builder copy work in #896. Check the GDAL and Zarr loader entry points and RasterRefImpl::band. Done means loaders return buffer, source_shape, and view, while RS_EnsureLoaded preserves the realized layout and non-identity views without rejecting them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100