lance-format / lance-format/lance

Register re-scannable write sources to avoid spilling re-readable inputs (Scanner/LanceDataset)

Open
#7,895 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Problem

The Python write path classifies sources into "materialized" (wrapped in an in-memory MemTable) vs. "streaming" (one-shot), via _is_materialized / _coerce_reader in python/python/lance/types.py.

Re-readable but not-in-memory sources — LanceDataset, pa.dataset.Dataset, pa.dataset.Scanner — are converted with .to_reader() and fall through to the streaming path. On the Rust side they arrive as a one-shot SendableRecordBatchStream of unknown size, so with conflict_retries > 0 they get drained into a memory/disk spill (spilling_table_provider) on every merge_insert — even though the source could simply be re-scanned. The re-readability is thrown away at the boundary.

Impact is limited to wasted work (an extra buffer/double-store on retrying merges), not correctness, but it's avoidable.

Proposed improvements (coupled)
  1. Re-scannable TableProvider. Add a provider that re-runs the source's scan on each .scan() — no spill, no full materialization — for sources that advertise re-readability (Scanner, LanceDataset, pa.dataset.Dataset). This eliminates both the spill and the double-store for those inputs.

  2. singledispatch source registry. Refactor the source classification/coercion (_is_materialized + _coerce_reader) into a functools.singledispatch registry (as done in LanceDB), returning the source plus a strategy (materialize / re-scannable / one-shot). This collapses the two hand-maintained type lists into one extensible registry and lets first- and third-party types register their own handling — e.g. a re-scannable registration wouldn't spill.

These are really one refactor: (2) is the mechanism, (1) is its first payoff.

Context

Follow-up from #7368 review (thanks @hamersaw for flagging both the type-list maintainability and the auto-inference of spill-on-retry).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in python/python/lance/types.py with _is_materialized and _coerce_reader, then trace how merge_insert handles conflict_retries and the Rust spilling_table_provider. Compare the Scanner, LanceDataset, and pa.dataset.Dataset paths with the existing materialized and streaming strategies. Done means re-readable sources are registered as re-scannable, avoid spill and double-store on retrying merges, and the source classification uses one extensible registry.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.