lance-format / lance-format/lance

bug: merge_insert analyze_plan reports the streaming plan for materialized sources

Open
#8,771 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Summary

MergeInsertBuilder.analyze_plan(data) coerces every input to a RecordBatchReader and then to a one-shot stream, so it reports the streaming plan even when execute(data) on the same input would run a different one.

Why the two plans differ

execute() routes a materialized source (pa.Table, pa.RecordBatch, a pandas or polars DataFrame, a dict, a list[dict]) through an in-memory table, which reports an exact row count and byte size. DataFusion's JoinSelection then collects the source as the hash join's build side and rewrites the join type. A stream reports no statistics at all, so nothing can be collected on that side and the target is collected instead.

So the side that gets buffered, and the join type, both depend on how the source was wrapped. analyze_plan reports the wrapping it chose rather than the one execute would choose.

Effect

A user profiling a merge reads the wrong side. The bytes_read on LanceRead and the output_rows on the join belong to a plan that is not the one their call runs.

The docstring example in dataset.py shows this directly: it passes a pa.table and prints a plan ending in StreamingTableExec, which is not what that input executes.

Where
  • python/python/lance/dataset.py: analyze_plan calls _coerce_reader and stops there, while execute dispatches on _is_materialized and calls execute_batches for materialized inputs.
  • python/src/dataset.rs: the analyze_plan binding converts to a reader and then to a stream.
  • rust/lance/src/dataset/write/merge_insert.rs: MergeInsertJob::analyze_plan takes a SendableRecordBatchStream and wraps it in one_shot_provider, with no entry point for materialized or re-scannable sources.
Note on explain_plan

explain_plan takes a schema rather than data, so it cannot tell which wrapping a caller would get and will keep reporting the streaming shape. Only analyze_plan, which receives the real source, can be made accurate.

Found while working on the source-statistics bullets of #4583.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read python/python/lance/dataset.py first, comparing analyze_plan with execute and its _is_materialized dispatch. Then trace the binding in python/src/dataset.rs and MergeInsertJob::analyze_plan in rust/lance/src/dataset/write/merge_insert.rs. Done means analyze_plan matches execute for materialized inputs while retaining the streaming behavior for streams; use the dataset.py docstring example to check the reported plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, rust
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.