feature: add lazy Polars integration for nearest-neighbor search

Related to #1507, pola-rs/polars#12389, and pola-rs/polars#29003

Problem

Polars can scan Lance through PyArrow with projection and supported scalar-filter pushdown, but it has no clear query-scoped API for Lance nearest-neighbor search.

polars-pylance and polars-lance validate demand and provide useful implementation evidence. polars-pylance currently supports nearest search with downstream postfiltering, but does not expose explicit source prefilter proposed here. Neither Lance nor Polars currently defines or maintains official integration interface with these semantics.

Required semantics

These operations must remain distinct:

• prefilter= restricts eligible rows before Lance computes top-k
• downstream .filter(...) filters ranked results afterward and may return fewer than k rows
• unsupported prefilters fail instead of silently becoming postfilters
• downstream filters are never implicitly reinterpreted as prefilters

Proposed API

Add optional Polars integration owned by Lance:

from lance.polars import scan_nearest

lf = scan_nearest(
    source,
    vector_column="embedding",
    query=query,
    k=10,
    metric="cosine",
    prefilter="category = 'docs'",
    use_index=True,
)

Proposed architecture

Draft Polars #29003 proposes LazyFrameResolver, which provides projection, optimizer-safe row limit, individual filters in both Polars and PyArrow forms, applied-filter acknowledgement, and resolver version keys.

Lance-owned resolver could:

• pin Lance dataset version for repeatable collection
• configure nearest search per query
• delegate explicit prefilter and index behavior to Lance
• preserve downstream Polars filters as postfilters
• reuse existing PyArrow/Substrait predicate pushdown where safe
• exchange batches through Arrow without adding Lance dependencies to Polars core

Current IO-plugin prototype remains useful as validation and fallback, but resolver is cleaner long-term integration point.

Initial scope

First implementation should include only:

• lazy scan_nearest
• explicit prefilter
• downstream postfilter
• indexed and exact search modes
• projection and safe limit pushdown
• version pinning
• repeated-collection and serialization support

Predicate-language expansion, ordinary scan_lance, writes, full-text search, Polars Cloud, and direct Rust integration should remain separate follow-ups.

Prototype result

Existing prototype using Polars IO plugin and Lance scanner verified:

• parity with direct Lance scanner
• distinct prefilter and postfilter behavior
• correct filter/limit ordering
• projection pushdown
• invalid-prefilter failure
• indexed plans containing ANN and scalar-index nodes

Maintainer decision

Would optional integration under lance.polars be acceptable, assuming Polars resolver direction is confirmed?

Start by reading the proposed Polars LazyFrameResolver direction in Polars #29003 and the existing IO-plugin prototype with the Lance scanner. Compare the prototype's prefilter, postfilter, projection, limit, indexing, and version-pinning behavior with the proposed lance.polars.scan_nearest entry point. Done means the maintainer accepts the resolver direction and the listed initial scope is supported without changing Polars core.