lance-format / lance-format/lance
Register re-scannable write sources to avoid spilling re-readable inputs (Scanner/LanceDataset)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Problem
The Python write path classifies sources into "materialized" (wrapped in an in-memory MemTable) vs. "streaming" (one-shot), via _is_materialized / _coerce_reader in python/python/lance/types.py.
Re-readable but not-in-memory sources — LanceDataset, pa.dataset.Dataset, pa.dataset.Scanner — are converted with .to_reader() and fall through to the streaming path. On the Rust side they arrive as a one-shot SendableRecordBatchStream of unknown size, so with conflict_retries > 0 they get drained into a memory/disk spill (spilling_table_provider) on every merge_insert — even though the source could simply be re-scanned. The re-readability is thrown away at the boundary.
Impact is limited to wasted work (an extra buffer/double-store on retrying merges), not correctness, but it's avoidable.
Proposed improvements (coupled)
-
Re-scannable
TableProvider. Add a provider that re-runs the source's scan on each.scan()— no spill, no full materialization — for sources that advertise re-readability (Scanner,LanceDataset,pa.dataset.Dataset). This eliminates both the spill and the double-store for those inputs. -
singledispatchsource registry. Refactor the source classification/coercion (_is_materialized+_coerce_reader) into afunctools.singledispatchregistry (as done in LanceDB), returning the source plus a strategy (materialize / re-scannable / one-shot). This collapses the two hand-maintained type lists into one extensible registry and lets first- and third-party types register their own handling — e.g. a re-scannable registration wouldn't spill.
These are really one refactor: (2) is the mechanism, (1) is its first payoff.
Context
Follow-up from #7368 review (thanks @hamersaw for flagging both the type-list maintainability and the auto-inference of spill-on-retry).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in python/python/lance/types.py with _is_materialized and _coerce_reader, then trace how merge_insert handles conflict_retries and the Rust spilling_table_provider. Compare the Scanner, LanceDataset, and pa.dataset.Dataset paths with the existing materialized and streaming strategies. Done means re-readable sources are registered as re-scannable, avoid spill and double-store on retrying merges, and the source classification uses one extensible registry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100