apache / apache/sedona-db

RS_EnsureLoaded / AsyncByteLoader: round-trip non-identity band views (view in -> realized view out)

Open
#897 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 4h
Merged PRs (30d)
90

Description

## Background

`RS_EnsureLoaded` materialises OutDb band bytes and passes InDb bands through. Today it cannot handle a band with a **non-identity view** (any slice / crop / broadcast / permutation, where the visible `shape()` differs from the backing `raw_source_shape()`):

- `ensure_loaded` rebuilds each band field-by-field via `start_band_nd` and never copies `band.view()`, so the view is silently dropped.
- `OutDbLoadRequest` carries only `source_shape` (the full backing extent); a loader has no way to learn that a sub-window was wanted.
- `AsyncByteLoader::load(req) -> Buffer` returns bare bytes; the caller can't tell what region/layout they represent.

This is **latent, not live**: `RasterRefImpl::band()` currently rejects non-identity views (`"view composition is not yet implemented"`), so such bands are unconstructible. The guardrail must come off as part of view-composition work (#813) — and when it does, `RS_EnsureLoaded` will silently corrupt viewed rasters unless this is addressed. **This issue is a correctness co-requisite of view composition landing.**

## The contract change: view in → realized view out

If the request carries a desired view (so a cloud loader *can* range-read just the sub-window instead of fetching a whole chunk), the loader has a legitimate choice, and the caller can't infer which it made from a bare buffer:

| Loader behavior | buffer | realized source_shape | realized view |
|---|---|---|---|
| Ignores crop (atomic chunk/tile — today's backends) | full backing bytes | full backing shape | input's view (unchanged) |
| Honors crop, returns contiguous cropped bytes | smaller | cropped shape | identity over cropped shape |
| Honors crop, returns full layout + selecting view | full | full | crop view |

`source_shape` and `view` move independently across these rows and neither is derivable from the buffer (its length only constrains `Π source_shape × byte_size`; a view entry doesn't reveal its source axis's true length). So the loader must report **all three** — the response is the data-bearing portion of a band:

```rust
struct LoadedBand {
buffer: Buffer,
source_shape: Vec, // natural C-order extent of `buffer`
view: Vec, // how visible axes map onto source_shape
}

async fn load(&self, req: &OutDbLoadRequest<'_>) -> Result;
```

`OutDbLoadRequest` gains the desired `view` (advisory — a loader may ignore it and return the full backing). Every loader, even the atomic ones, returns `source_shape` + an echoed/identity view, not just bytes.

## Seam with the builder copy work (#896)

With `copy_band_from` (#896), `RS_EnsureLoaded` becomes copy-the-metadata, override the loader-controlled layout:

```rust
copy_band_from(input) // name, dim_names, dtype, nodata, outdb_uri/format
.with_data_layout(resp.buffer, resp.source_shape, resp.view);
```

The loader owns exactly the three things a partial read can change (`buffer`, `source_shape`, `view`); the builder copy owns everything else. "Ignored the crop" is just the special case where the realized view/shape equal the input's.

## Scope

- `OutDbLoadRequest`: add the desired `view`.
- `AsyncByteLoader::load`: return `LoadedBand { buffer, source_shape, view }` instead of `Buffer`; update GDAL + Zarr loaders to echo `source_shape` + identity view (they ignore the crop for now).
- `RS_EnsureLoaded`: carry the input band's view into the request; build output bands from the realized response (via #896's copy + layout override).
- Remove the `RasterRefImpl::band()` non-identity-view rejection as the view-composition machinery (#813) makes them constructible (coordinate with that workstream).

## Not in scope

- Actually *implementing* crop pushdown in any loader (cloud range-reads). This issue only ensures the request/response can express it and that `RS_EnsureLoaded` stays correct when views exist.

## Related

- #813 (view composition / `RasterBuilder::with_view` — the forcing function; co-requisite)
- #896 (RasterBuilder copy APIs — the mechanism for the round-trip)
- #894 (zero-copy band-data path)

Contributor guide

Open the contributing guide

Research direction

Start with OutDbLoadRequest, AsyncByteLoader::load, and RS_EnsureLoaded, then review the related view-composition work in #813 and builder copy work in #896. Check the GDAL and Zarr loader entry points and RasterRefImpl::band. Done means loaders return buffer, source_shape, and view, while RS_EnsureLoaded preserves the realized layout and non-identity views without rejecting them.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.