lance-format / lance-format/lance
bug: merge_insert analyze_plan reports the streaming plan for materialized sources
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Summary
MergeInsertBuilder.analyze_plan(data) coerces every input to a RecordBatchReader and then to a one-shot stream, so it reports the streaming plan even when execute(data) on the same input would run a different one.
Why the two plans differ
execute() routes a materialized source (pa.Table, pa.RecordBatch, a pandas or polars DataFrame, a dict, a list[dict]) through an in-memory table, which reports an exact row count and byte size. DataFusion's JoinSelection then collects the source as the hash join's build side and rewrites the join type. A stream reports no statistics at all, so nothing can be collected on that side and the target is collected instead.
So the side that gets buffered, and the join type, both depend on how the source was wrapped. analyze_plan reports the wrapping it chose rather than the one execute would choose.
Effect
A user profiling a merge reads the wrong side. The bytes_read on LanceRead and the output_rows on the join belong to a plan that is not the one their call runs.
The docstring example in dataset.py shows this directly: it passes a pa.table and prints a plan ending in StreamingTableExec, which is not what that input executes.
Where
python/python/lance/dataset.py:analyze_plancalls_coerce_readerand stops there, whileexecutedispatches on_is_materializedand callsexecute_batchesfor materialized inputs.python/src/dataset.rs: theanalyze_planbinding converts to a reader and then to a stream.rust/lance/src/dataset/write/merge_insert.rs:MergeInsertJob::analyze_plantakes aSendableRecordBatchStreamand wraps it inone_shot_provider, with no entry point for materialized or re-scannable sources.
Note on explain_plan
explain_plan takes a schema rather than data, so it cannot tell which wrapping a caller would get and will keep reporting the streaming shape. Only analyze_plan, which receives the real source, can be made accurate.
Found while working on the source-statistics bullets of #4583.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read python/python/lance/dataset.py first, comparing analyze_plan with execute and its _is_materialized dispatch. Then trace the binding in python/src/dataset.rs and MergeInsertJob::analyze_plan in rust/lance/src/dataset/write/merge_insert.rs. Done means analyze_plan matches execute for materialized inputs while retaining the streaming behavior for streams; use the dataset.py docstring example to check the reported plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, rust
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100