perf: KeepFiniteVectors walks every row even when the whole batch is finite

LABEL: performance

Description

KeepFiniteVectors::transform in rust/lance-index/src/vector/transform.rs decides which rows to keep by iterating the vector column row by row:

data.iter().enumerate().for_each(|(idx, arr)| {
    if let Some(data) = arr { /* per-row finiteness check */ }
});

FixedSizeListArray::iter() yields an Option<ArrayRef> per row, so this allocates one ArrayRef per vector before checking it. The transform runs on every batch of every IVF index build (new_ivf_transformer_with_quantizer pushes it for the flat, PQ, SQ and RQ paths), and the overwhelmingly common case is that no value is non-finite — in which case all that work ends with valid.len() == batch.num_rows() and the batch returned unchanged.

The same question can be answered with a single pass over the flattened values buffer, which needs no per-row allocation.

Measured on 100,000 rows of 128-dimensional f32, all finite, release build, timing only transform: 8.59 ms per call as it stands, 5.53 ms with a flat-buffer pre-check, so about 3.05 ms saved per batch of that size.

Expected behavior

Answer "is anything non-finite in this batch" once over the values buffer, and fall back to the per-row walk only when something is.

Lance version

13.0.0-beta.4 (main)

Language binding

Rust

Start in rust/lance-index/src/vector/transform.rs at KeepFiniteVectors::transform and review how the flattened values buffer relates to the current FixedSizeListArray row iteration. Add the batch-level finite check and retain the row walk only for batches containing non-finite values; done means finite batches are returned unchanged while invalid rows are still filtered correctly.