Follow-ups to #6985: further *View-type support and optimization

Background

#6985 added support for Arrow Utf8View / BinaryView by storing them as the classic offset-based Utf8 / Binary on disk. View columns are downcast to their base types in SchemaAdapter (lance/src/dataset/utils.rs, via physical_field / downcast_view_columns), with a fallback arrow_cast in DataBlock::from_arrays (lance-encoding/src/data.rs) for non-stream write paths. This follows the direction in #5172: Lance does not introduce view types into the on-disk format; the physical type stays offset-based.

That PR left several adjacent items open. This issue collects them as independently pickable tasks, each with benefit, rough cost, and risk, so we can decide which (if any) are worth doing. None of these change the on-disk format — they only affect conversion at the Arrow boundary.

Guiding observation: the two directions are asymmetric

• Write (view → offset) requires a gather: a view array's bytes are scattered across data buffers (short strings are inlined in the 16-byte views), while the on-disk block (VariableWidthBlock) is one contiguous buffer + offsets. Materializing contiguous bytes is unavoidable.
• Read (offset → view) is cheap: the on-disk block is already one contiguous buffer, which a view array can reference; only the per-row views array has to be built (no bulk copy).

This asymmetry is why the read-side task is more attractive than the write-side one.

---

Task 1 — Direct Utf8View/BinaryView → block encoding on write (skip arrow_cast)

Summary. Replace the arrow_cast::cast(Utf8View → Utf8) with a routine that builds the VariableWidthBlock directly from a StringViewArray / BinaryViewArray in a single pass. (Utf8View/BinaryView are leaf/variable-width types handled in DataBlock::from_arrays.)

Benefit. The gather is required regardless, so this only removes work around it: an intermediate StringArray, a second offsets pass (arrow_cast builds Arrow offsets, then stitch_offsets re-derives Lance offsets), and — when multiple input arrays are combined — a redundant byte copy in concat_into_one. Expected end-to-end write improvement is small (a small fraction of conversion cost, which is itself behind encoding/compression/IO). Most visible for columns of very short (inlined) strings.

Implementation.

• A single-pass string_view / binary_view → VariableWidthBlock builder in data.rs (i32/i64 offsets, nulls, multiple input arrays).
• To reach the hot path, SchemaAdapter would have to stop downcasting the view data while the recorded schema stays base-typed — which inverts the purpose of SchemaAdapter (it reconciles the input batch to the physical schema) and requires the writer's schema validation / encoder dispatch to accept a view array for a base-typed field. Changing only the from_arrays fallback arm yields no hot-path benefit, since SchemaAdapter already casts upstream.

Rough size / risk. The builder is self-contained in data.rs. Wiring it onto the hot path is the larger, riskier part (the SchemaAdapter contract change + schema-consistency auditing + tests).

Lower-risk alternative. Keep the SchemaAdapter contract and just replace its arrow_cast with a hand-rolled single-pass view→base builder — same output, no contract change, captures whatever margin exists over arrow_cast.

Recommendation. Low priority; pursue only if profiling shows the cast is hot, and prefer the lower-risk alternative.

---

Task 2 — Optional Utf8View/BinaryView decoding on read

Summary. Let the reader materialize Utf8View / BinaryView arrays directly from the offset-based on-disk block, instead of always producing Utf8 / Binary. A reader-API capability driven by the requested output schema, not a default.

Where the benefit applies. Lance's decoder produces Arrow that is consumed by many engines and bindings — Lance's own DataFusion-based execution, the Python/PyArrow bindings, the LanceDB SDK, anything via the Arrow C Data Interface, and external query engines (e.g. Spark, Ray). The benefit exists only when view-exploiting compute is co-located with the read — e.g. a pushed-down DataFusion subplan doing string filtering/comparison/slicing — where decoding straight to views avoids a Utf8 → Utf8View cast and lets those operators use the view layout. (#6985 notes DataFusion 43+ produces view types from string functions.) When Lance is a pure Arrow source for an external engine, the consumer chooses its own representation; if it wants Utf8, handing it views forces a view → base conversion downstream (the gather). So the win is concentrated in the pushdown/co-located case and is neutral-to-negative for storage-to-external-engine handoff.

Even where it applies, magnitude is modest: the per-row view build is unavoidable whether the reader or a post-read cast does it, so the saving is a redundant pass plus a plan node, not a fundamental speedup.

How the consumer selects view vs. base — via the requested schema, not a feature flag, and not a Lance default. The stored column is Utf8; the desired output type is expressed through the output (projected) schema the reader is asked to produce. If that schema types a stored-Utf8 column as Utf8View, the reader decodes into views; otherwise it produces Utf8, as today. This is preferable to a global flag because it is per-column and per-read — a single plan can request views only on the columns a view-friendly operator will touch — and it is set by whichever consumer benefits, exposed as a reader-API capability that each integration (the DataFusion scan, or the direct Arrow reader behind the SDK / PyArrow / external connectors) opts into independently. The reader would recognize a restricted set of view↔base aliases (Utf8↔Utf8View, Binary↔BinaryView) in the requested schema, not arbitrary cast-on-read.

Why it must be requested, never the default. Producing views unconditionally would make any base-type consumer pay view → base (the gather). Most Arrow consumers expect the classic offset layout, so defaulting to views would create that costly direction for them.

Implementation.

• A view branch in VariableWidthBlock::into_arrow that builds a StringViewArray / BinaryViewArray from offsets + data (inline-vs-reference logic, i32/i64 offsets, nulls).
• Read-path plumbing to honor a view-typed request for a base-typed stored column, exposed at the reader API so each integration can opt in.

Rough size / risk. Decode branch is small; the larger piece is the requested-schema plumbing in the projection / read-schema layer. No on-disk change, no write-path contract inversion. Low risk.

Recommendation. Worthwhile mainly for view-friendly compute pushed down into Lance's own execution; drive it from the requested output schema and keep base types the default for external-engine consumers.

---

Task 3 — Support ListView / LargeListView

Summary. Add support for the Arrow ListView / LargeListView types by storing them as the offset-based List / LargeList on disk — the same interception approach #6985 used for Utf8View / BinaryView.

Current state (unsupported).

• The schema layer (LogicalType::try_from / Field::try_from in lance-core/.../datatypes/field.rs) maps List / LargeList but has no ListView / LargeListView case.
• SchemaAdapter::physical_field downcasts only Utf8View / BinaryView, not list-views.
• DataBlock::from_arrays lists ListView / LargeListView in its unsupported/panic! arm (alongside List / LargeList, which are handled earlier by the logical list encoder rather than from_arrays).

So a column of these types cannot currently be written.

Benefit. Removes an unsupported-type failure for inputs Arrow and engines can legitimately produce, with no new on-disk representation — symmetric with the string/binary-view support already shipped.

Implementation (parallels #6985). Intercept and downcast to the offset-based list, so the existing list encoder handles the rest:

• physical_field: map ListView → List, LargeListView → LargeList; downcast_view_columns casts the column.
• Schema layer (field.rs): map the two view types to List / LargeList so the declared schema is accepted.
• No new from_arrays arm is needed once the column is downcast upstream (list types are decomposed by the logical list encoder before reaching from_arrays); optionally add a from_arrays fallback arm for non-stream paths, mirroring #6985.
• Verify arrow_cast supports ListView → List / LargeListView → LargeList; since list-views permit overlapping / out-of-order child ranges, the conversion must compact/reorder the child — if arrow_cast doesn't cover it, a manual compaction is required.
• Filter coercion (safe_coerce_scalar): review whether list-view literals can appear; likely not applicable.

Rough size / risk. Small-to-moderate, mirroring #6985: a few schema / physical_field arms + tests. Main open question is arrow_cast coverage for the list-view→list compaction.

Recommendation. Worth doing for type-coverage / robustness; the most concrete correctness item here.

---

Task 4 — Nested *View types inside Struct / List / Map / FixedSizeList

Summary. SchemaAdapter downcasts only top-level fields, so view types nested inside containers are not converted and reach the encoders as views.

Current state. Nested cases hit todo!() / unimplemented!() / panic!:

• lance-encoding/.../logical/struct.rs — todo!() building a decoder for a nested ListView / LargeListView field (likely unreachable in practice, since stored fields are never list-view once downcast — worth confirming).
• lance-encoding/.../logical/fixed_size_list.rs — unimplemented!() for FixedSizeList<ListView> garbage filtering.
• DataBlock::from_arrays — view / list-view leaves reached via nested recursion (e.g. a struct or FSL containing a view child) hit the unsupported arm.

Benefit. Correctness for nested view columns, which currently fail. The triggering combinations (e.g. FixedSizeList<ListView>, a Struct with a nested view child) are exotic and rarely produced in practice, so this is lower priority.

Rough size / risk. Medium; requires extending the downcast/cast handling from top-level into nested traversal. Pre-existing gap, independent of Tasks 1–3, though naturally bundled with Task 3 since top-level-only downcast is what leaves nested types exposed.

---

Summary

Task	Direction	Fundamental cost	Benefit	Risk
1. Direct view→block encode	write	gather (unavoidable)	small write-CPU saving	hot-path needs SchemaAdapter contract change (medium)
2. Optional view decode	read	none (data buffer reused; per-row views built)	avoids a cast for view-friendly compute pushed down into Lance's execution	low
3. ListView / LargeListView support	write	gather on cast-down	type coverage / removes an unsupported-type failure	low–medium (pending arrow_cast coverage)
4. Nested *View in containers	both	—	correctness for nested view columns	medium; pre-existing, exotic

Suggested priority: Task 3 (concrete correctness), then Task 2 if view-friendly pushdown workloads warrant it; Tasks 1 and 4 are lower priority (small/risky and exotic, respectively).

Start with Task 3 and read the type mappings in lance-core/.../datatypes/field.rs, SchemaAdapter in lance/src/dataset/utils.rs, and the conversion entry point in lance-encoding/src/data.rs. Check whether arrow_cast supports ListView to List compaction, then add the schema and conversion coverage needed for ListView and LargeListView. Done means these types can be written through the existing offset-based encoders without changing the on-disk format.