apache / apache/sedona-db

RS_EnsureContiguous UDF: materialize strided band views to contiguous bytes via an explicit plan node

Open
#899 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 4h
Merged PRs (30d)
90

Description

## Problem

Non-identity band views (slice-with-step, broadcast, transpose, inner-axis slice) produce a *strided* byte layout that isn't packed row-major, so kernels needing packed bytes can't consume them directly. #813 establishes that the contiguous accessor (`NdBuffer::as_contiguous()`) returns bytes zero-copy when the layout is already packed and **errors** otherwise — it never silently materializes — and the GDAL bridge rejects strided views. Materialization is intentionally kept *out* of the read path: it belongs in an explicit, CSE-dedupable plan node. This issue adds that node.

## Proposal

Add `RS_EnsureContiguous`, mirroring `RS_EnsureLoaded` (#886):

- A UDF that takes a raster and returns one whose bands are identity-view, packed row-major — strided views materialized, already-contiguous bands passed through zero-copy.
- A logical optimizer rule (in `sedona-query-planner`) that wraps the raster args of kernels annotated `needs_contiguous`, using the `SedonaScalarUDF` metadata map (the `NEEDS_PIXELS_METADATA_KEY` mechanism generalizes to a `needs_contiguous` key).
- The materialization primitive uses caller-managed scratch (or a visitor) reused across rows — e.g. `NdBuffer::materialize_into(&mut Vec)` — never a per-row allocation. This is where the review thread on `array.rs:288` (externally-managed scratch / visitor) is addressed.
- Fast path: `NdBuffer::is_contiguous()` (added in #813) makes `RS_EnsureContiguous` a zero-copy no-op for rows whose layout is already packed — including contiguous *non-identity* views like outer-axis slices, not just the strict identity view.

## Composition with `RS_EnsureLoaded`

`RS_EnsureContiguous` needs bytes present, so it must run *after* load: `RS_X(RS_EnsureContiguous(RS_EnsureLoaded(rast)))`. Prefer unifying both into one metadata-driven normalization rule that injects the correct nested wrappers based on a kernel's flags, rather than two independent rules.

## Knock-on cleanups

- The GDAL bridge's strided-view rejection (#813) can relax once the plan guarantees identity-view inputs.
- Re-introduce the strided-copy helper removed in #813 (`materialize_strided`) here, as the scratch/visitor primitive.

## Design tenet

Materialization is never a transparent side effect of a trait accessor — the `BandRef` byte surface is `nd_buffer()` (zero-copy) plus `NdBuffer::as_contiguous()` (borrow-or-error). This UDF is the explicit, plan-visible place where strided→packed copying happens.

## Related

- #886 (`RS_EnsureLoaded` — the pattern this mirrors)
- #897 (`RS_EnsureLoaded` view round-trip — `EnsureLoaded` preserves the view; `EnsureContiguous` collapses it)
- #813 (view machinery — adds `is_contiguous`/`as_contiguous`, rejects strided reads)

Contributor guide

Open the contributing guide

Research direction

Start by reading #886 for the RS_EnsureLoaded pattern, then inspect the planner in sedona-query-planner and the scratch/visitor discussion around array.rs:288. Compare the view and contiguous-buffer behavior from #813 and #897. Done means the explicit RS_EnsureContiguous UDF and metadata-driven normalization handle loaded strided inputs, reuse scratch across rows, and preserve zero-copy behavior for contiguous bands.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.